Benchmarking 28 OCR models for Entropia Universe text parsing

Summary

EntropiaOrme reads the game’s on-screen data with an OCR model to feed its analytics. By benchmarking 28 OCR models against images taken from the game’s UI, OpenOCR SVTRv2-mobile was chosen: an Apache-2.0 model of about 24 MB that runs at about 3.5 ms per image, with 100% effective accuracy for this task. The evaluation dataset, harness, and full per-model results across all the tested hardware are in the public benchmark repository.

Problem statement

Entropia Universe outputs most of the data needed to run analytics to a log file (chat.log). Some values, however, are only present in the game UI; therefore, this article covers a study on using OCR models to parse that data. Here are some examples of the type of data needing to be processed by OCR:

Methodology

The goal is finding the lightest model that reliably parses the game UI. To measure that tradeoff, 28 models ranging from small recognisers to large vision-language models were selected for testing.

The test set consists of 594 text and numeric images from the game UI with human-verified labels. For text values, the vocabulary is closed, which means that slightly misread names can be fuzzy matched (“Aglity” to “Agility”). Therefore, “effective accuracy” is reported (percentage of correct/recoverable predictions after postprocessing). Each model was also timed per image, with the reported latency measured on an Nvidia A1000 machine.¹

Results

Accuracy and latency are plotted together below, with the desirable corner at the top left; the same figures are available as a table.

The primary finding is lightweight models being both the fastest and most accurate, therefore not being a compromise for this simple task. The heavier models bought neither accuracy nor speed, where the dominant failure mode is domain mismatch. Model size was not a predictor of accuracy on this clean, small-text task.

Two models are viable to bundle: OpenOCR SVTRv2 and RapidOCR, both at 100% effective accuracy under an Apache-2.0 licence. Surya, the third at 100%, is CUDA-only. With accuracy settled between the remaining two, the choice becomes a latency question, measured across every configuration.

When processing images serially, RapidOCR on the CPU is the quicker of the two. When batched, OpenOCR SVTRv2 on the GPU reads about 286 cells per second against RapidOCR’s 142, roughly twice the throughput.

Conclusion

EntropiaOrme reads a panel’s cells in a batch, and batched on the GPU OpenOCR SVTRv2 is the fastest option at about 3.5 ms per image. Further, Entropia Universe is a CPU-bound game, which makes running the OCR model on GPU more desirable so as not to starve the game. Therefore, OpenOCR SVTRv2 is the one EntropiaOrme bundles.