Testing the competition

This page is written and maintained by Mamadou DIOP. Please contact him if you have any question or clarification about your application.

I’m using Samsung Galaxy S10+ and have installed the latest version (June 26, 2021) for each application.

I have just concatenated the videos with no editing. You can contact me if you need the raw videos (rushes).

Please keep in mind that unlike the competition our implementation is available for testing in unconstrained environment which means you can use images at any resolution and the face can be at any distance from the camera lens.

Next video#0 shows a stress test on our implementation using different type of attacks:



FaceTEC

Web site: https://www.facetec.com

I have to give credit to these guys: They are very good in marketing.

They have disabled screen shot/screen recording on the liveness screen which makes it impossible to record a video like what I did for other applications.

<1% FRR

They claim on their website that the FRR (False Rejection Rate) is less than 1%. That’s clearly not true. Using a Galaxy S10+ (high-end camera) or Wiko View4 (low-end camera) it was a nightmare to enroll a genuine face (myself). The app keeps asking me to “Move away”, “Move closer”, “Hold Steady”… and at the end I get a message asking me to try again. Sometimes I have to try 10 to 15 times in order to enroll a genuine face. The comments on Google Play (back to 2017) show multiple users have the same issue. Tried in different rooms with very good lighting but had the same issues. The experience is different on iPhone12 Pro: It’s fast and accurate.

1/12,800,000 FAR

This claim may be true but comes with a cost (low FRR). Again, it’s easy to achieve a good FAR if you accept to sacrifice your FRR to the marketing gods. My next code has a perfect FAR (0/1,000,000,000,000):

def liveness(image):
    return False # Always say it's spoof

Liveness detection in 2 seconds

The network transfer alone to send the video selfie takes way more than 2 seconds (used with very good 4G network). In my experience (Galaxy S10+) it takes at least 15 seconds, sometimes it’s several minutes. On iPhone 12 Pro it’s way faster.

$100,000 Spoof Bounty Program

That’s clearly a marketing message. I have managed to bypass the liveness check using several 3D masks. As already explained, screen recording is disabled on their app which made it impossible to capture.

800% improvements

I’ve subscribed to their mailing list and few weeks ago they sent a mail saying their accuracy has improved by 800%. You can also find the same information here. No comment.


BioID

Web site: https://www.bioid.com/

Despite being ISO/IEC 30107 (level 1 and 2) certified the implementation can be easily fooled. I have managed to bypass (enroll and verify) the liveness detector using Replay-Attacks (video) and 3D face masks. I have also managed to enroll and verify myself while hiding almost 80% of my face with dark glasses and a COVID mask (check the video).

The next video#1 shows the results:



Huawei

Web site: https://developer.huawei.com/consumer/en/doc/development/HMSCore-Guides-V5/liveness-detection-0000001051386243-V5

I have developed and released an Open Source project on Github in order to test this implementation: https://github.com/DoubangoTelecom/HuaweiFaceLiveness

Months ago I tested this application against Print-Attacks, you can retrieve the video at https://youtu.be/52W2K5yJIl4. This time I wanted to test 3D attacks. The application fails to detect these attacks but I think it’s normal as they’re not branding it as a 3D liveness detector.

Check next video#2 video (3D attacks):



Wiko

I have used a View 4 phone to register my face. Surprisingly, it was very easy to unlock the phone using a high definition picture of myself.


On the video I’m wearing dark glasses and a face mask to make sure I’m not unlocking the phone by inadvertence.


Samsung

Like Wiko I registered my face (Galaxy S10+) and tried to unlock the phone using a picture of myself but … I failed. I can’t say if their implementation is really strong or not because they disable the face recognition when you fail 3 (or 5) times. Disabling face recognition for a period of time and forcing you to enter your password make it painful to try multiple times to find the right angle and lighting.


Sybrin

Web site: https://corporate.sybrin.com/

The application keeps crashing after OCR’ing the passport. So, haven’t managed to test their liveness detector.



Onfido

Web site: https://onfido.com

They have a very light and basic active liveness detector. I don’t know if the one in their application on Google Play is the same as what they provide to their customers but it looks weak and easy to break. This implementation fails on 3D and replay attacks.



Jumio

Web site: https://www.jumio.com

You have to successfully provide an identity document in order to test the liveness detector. I haven’t managed to pass the OCR. Tried with a real French passport and identity card by they fail to recognize the MRZ lines.

Next video#3 shows a try:


By the way, we (Doubango AI) have an MRZ reader and you can see it at work here:



Minivision-ai (turingtech.vip)

Web site: https://github.com/minivision-ai/Silent-Face-Anti-Spoofing

This is an open source implementation with a friendly commercial license (Apache). They do not claim to be a 3D liveness detector but the deep learning model suggests it is (Sofmax dense 3).

I managed to build the project and did some tests but it was clearly not good. On their home page they claim that they have a better version but it’s private. While browsing their Github issue tracker I found a comment about a better implementation. This implementation is from turingtech (https://turingtech.vip) and can be found at https://github.com/turingtech-vip/Face-Anti-Spoofing-Android-iOS. I unziped the APK and scanned the binaries and surprise surprise… it’s based on Minivision-ai. The implementation from turingtech is clearly more accurate which suggest they re-trained the model with more data.

This implementation is very sensible to lighting and easily fail to all attacks: Screen display, Paper photo, video replay, 3D masks…