Benchmark

It’s easy to assert that our implementation is fast without backing our claim with numbers and source code freely available to everyone to check.

Rules:

  • We’re running the processing function within a loop for #100 times.

  • The positive rate defines the percentage of images with at least #2 MRZ lines. For example, 20% positives means we will have #80 negative images (no MRZ lines) and #20 positives (with MRZ lines) out of the #100 total images. This percentage is important as it allows timing both the detector and recognizer.

  • We’re using high accuracy for the segmenter and bilinear interpolation.

  • All positive images contain a least #2 MRZ lines.

  • The initialization is done outside the loop.

The concept of negative and positive images is very important because in most use cases you’ll:

    1. Start the application

    1. Move the application to the MRZ zone to recognize the data

You only need a single “good frame” to recognize the MRZ lines. But, between step #1 and step #2 the application has probably processed more than #200 frames (40fps * 5sec). So, in such scenario the application have to process #201 frames before reaching the “good frame”: #200 negatives and #1 positive. If processing negative frame is very slow then, the application won’t be able to catch this “good frame” at the right time. A slow application will do one of these strategies:

    1. Drop frames to keep the impression that the application is running at realtime: In such scenario the positive frames will most likely be dropped (probability = 1/201 = 0.49%) which means reporting the time when this single “good frame” is caught.

    1. Enqueue the frames and process them at the application speed: This is the worse solution because you could run out of memory and when the application is running slowly then, you can spend several minutes before reaching this single “good frame”.

A fast application will run at 40fps and catch this “good frame” as soon as it’s presented for processing. This offers a nice user experience.

0% positives

20% positives

50% positives

70% positives

100% positives

Core i7-4790K
(Windows 8)

877 ms
114 fps

1975 ms
50.61 fps

3736 ms
26.76 fps

4901 ms
20.40 fps

6526 ms
15.32 fps

iPhone7
(iOS 13)

1990 ms
50.23 fps

4325 ms
23.11 fps

7982 ms
12.52 fps

10595 ms
9.43 fps

14201 ms
7.04 fps

Galaxy S10+
(Android 10)

2825 ms
35.39 fps

7575 ms
13.20 fps

12960 ms
7.71 fps

17636 ms
5.67 fps

21069 ms
4.74 fps

Raspberry Pi 4
(Raspbian OS)

4335 ms
23.06 fps

13555 ms
7.37 fps

27878 ms
3.58 fps

37399 ms
2.67 fps

47797 ms
2.09 fps

More information on how to build and use the application could be found at https://github.com/DoubangoTelecom/ultimateMRZ-SDK/blob/master/samples/c++/benchmark/README.md.

For Android and iOS the Benchmark application is in samples/android/benchmark and samples/ios/benchmark folders.

Please note that even if Raspberry Pi 4 have a 64-bit CPU Raspbian OS uses a 32-bit kernel which means we’re loosing many SIMD optimizations.