Benchmark

It’s easy to assert that our implementation is fast without backing our claim with numbers and source code freely available to everyone to check.

Rules:

  • We’re running the processing function within a loop for #100 times.

  • The positive rate defines the percentage of images with a valid credit card. For example, 20% positives means we will have #80 negative images (no credit card) and #20 positives (with credit cards) out of the #100 total images. This percentage is important as it allows timing both the detector and recognizer.

  • All positive images contain a single credit card.

  • The rectification layer is disabled.

  • The initialization is done outside the loop.

The concept of negative and positive images is very important because in most use cases you’ll:

  1. Start the application.

  2. Move the application to the credit card to recognize the data.

You only need a single “good frame” to recognize the credit card. But, between step #1 and step #2 the application has probably processed more than #200 frames (40fps * 5sec). So, in such scenario the application have to process #201 frames before reaching the “good frame”: #200 negatives and #1 positive. If processing negative frame is very slow then, the application won’t be able to catch this “good frame” at the right time. A slow application will do one of these strategies:

  1. Drop frames to keep the impression that the application is running at realtime: In such scenario the positive frames will most likely be dropped (probability = 1/201 = 0.49%) which means reporting the time when this single “good frame” is caught.

  2. Enqueue the frames and process them at the application speed: This is the worse solution because you could run out of memory and when the application is running slowly then, you can spend several minutes before reaching this single “good frame”.

A fast application will run at 40fps and catch this “good frame” as soon as it’s presented for processing. This offers a nice user experience.

0% positives

20% positives

50% positives

70% positives

100% positives

Xeon E31230v5 GTX 1070
(Ubuntu 18)

976 ms
102.43 fps

1602 ms
62.39 fps

2567 ms
38.94 fps

3214 ms
31.11 fps

4142 ms
24.13 fps

i7-4790K
(Windows 7)

4459 ms
22.42 fps

8811 ms
11.34 fps

15827 ms
6.31 fps

20571 ms
4.86 fps

27361 ms
3.65 fps

i7-4770HQ
(Windows 10)

8653 ms
11.55 fps

16583 ms
6.03 fps

29668 ms
3.37 fps

38636 ms
2.58 fps

52945 ms
1.88 fps

Galaxy S10+
(Android 10)

3673 ms
27.22 fps

13053 ms
7.66 fps

28304 ms
3.53 fps

38175 ms
2.61 fps

52685 ms
1.89 fps

Raspberry Pi 4
(Raspbian OS)

9731 ms
10.21 fps

35449 ms
2.82 fps

74698 ms
1.33 fps

101665 ms
0.98 fps

128269 ms
0.77 fps

More information on how to build and use the application could be found at https://github.com/DoubangoTelecom/ultimateCreditcard-SDK/blob/master/samples/c++/benchmark/README.md. For Android the Benchmark application is at samples/android/benchmark.

Please note that even if Raspberry Pi 4 have a 64-bit CPU Raspbian OS uses a 32-bit kernel which means we’re loosing many SIMD optimizations.