Stream injection (virtual camera) detection

We can detect all injection types. It exists multiple injection types but only 3 categories: image, video and deepfake.

  1. Image injection: The virtual camera reads a local image (png, jpeg, bmp…) and transmit it as video stream (looping).

  2. Video injection: The virtual camera reads a dynamic source and transmit it as video stream. The source could be local file (mp4, webm…), a window, a desktop, a presentation…

  3. Deepfake injection: An existing video stream (e.g. from a real camera) is transformed by a virtual camera to replace the faces with deepfakes.

The SDK was tested against multiple virtual Cameras (OBS Studio, ManyCam, v4l2loopback, Unity Virtual Camera…) and platforms (Windows, macOS, Linux, Android, iOS…) with high accuracy.

The next video shows the SDK catching stream injection using OBS Virtual Camera and Face2Face as explained here:

Smart-pass

The stereo image is very sensitive to noise which may lead to an injection score slightly lower than the genuine threshold without being “injected” (spoof). The smart pass module can sort out these edge cases.

This module is enabled using configuration entry inject_smartpass_enabled.

Aggressive mode

The stream injection detection algorithm requires camera intrinsics (focal length, optical center…) parameters computed from the stereo image. Unfortunately some Camera API providers alter the images coming from the lens with some post-processings making the intrinsics invalid. This was noticed with OpenCV and may be the case with others (FFMpeg, Gstreamer…). The aggressive mode requires low delay/motion in the stereo image and could increase the FRR (False Rejection Rate) which adds friction. To avoid adding friction in your app we recommend (on client side) to:

  • enable this mode if you’re using OpenCV, FFmpeg, Gstreamer…

  • disable this mode if you’re using a web browser (GetUserMedia), Android native API (e.g. Camera2) or iOS native API (e.g. AVFoundation).

This mode will flag any stereo image with inconsistent intrinsics (focal length, optical center…) as coming from a virtual camera. You can toggle it using our online demo:

A stereo image rejected because of this mode will be tagged with warning message stereo impaired(1) or stereo impaired(4).

Testing

In the next sections we’ll be using OBS Virtual Camera as it’s free and harder to detect than other virtual cameras.

Deepface injection

There are two ways to inject a deepfake into an application (e.g. zoom):

  1. Hook a real camera, change the face, display the result on a window, then capture the window on OBS Studio.

  2. Or, hook a real camera, change the face, send the result to a virtual camera, then use OBS Studio to read from the vcam.

The second method is more flexible and allow injecting high quality deepfakes. We use pyvirtualcam to hook the real camera, change the face using face2face and send the result to a Unity Virtual Camera. The source code is at below. OBS Studio is configured with Unity Virtual Camera as source. To check the effectiveness of this method, we tried it against https://faceapi.regulaforensics.com and easily managed to fool both their active and passive liveness implementations.

The source code for the second method is:

import onnxruntime
import numpy as np
import pyvirtualcam, cv2
from face2face import Face2Face

REAL_CAMERA_INDEX = 0 # images from this real camera will be sent
VIRTUAL_CAMERA_NAME = 'Unity Video Capture' # to this virtual camera (after deepfake superimposition)
FRAME_WIDTH = 1280
FRAME_HEIGHT = 720
FRAME_FPS = 15
DEEFACE_PATH = 'D:/inject_images/Obama.jpg' # image from https://en.wikipedia.org/wiki/Barack_Obama#/media/File:Official_portrait_of_Barack_Obama.jpg

with pyvirtualcam.Camera(width=FRAME_WIDTH, height=FRAME_HEIGHT, fps=FRAME_FPS, device=VIRTUAL_CAMERA_NAME) as cam:
    print(f'Using virtual camera: {cam.device}')
    onnxruntime.preload_dlls() # to avoid error 126 on "onnxruntime_providers_cuda.dll" at load time
    f2f = Face2Face(device_id=0) # set GPU device id, only if you have a GPU
    faces = f2f.detect_faces(DEEFACE_PATH)
    cap = cv2.VideoCapture(REAL_CAMERA_INDEX, cv2.CAP_DSHOW)
    cap.set(cv2.CAP_PROP_FORMAT, -1)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, FRAME_WIDTH)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT)
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        img = f2f.swap(frame, faces, enhance_face_model='gpen_bfr_512')
        frame = cv2.cvtColor(img if isinstance(img, np.ndarray) else frame, cv2.COLOR_BGR2RGB)

        cam.send(frame)
        cam.sleep_until_next_frame()

    cap.release()
    cv2.destroyAllWindows()

Video injection

Various video formats (raw, h264, vp8…) were tested using OBS Studio.

You can use genuine faces using videos from https://www.kaggle.com/datasets/trainingdatapro/ibeta-level-1-liveness-detection-dataset-part-1

Image injection

Various image formats (png, tif, jpeg…) were tested using OBS Studio: raw, lossly compression, lossless compression…

Android emulator

A video stream from an Android emulator (e.g. BlueStaks) will always be rejected by our stream injection detector as being from a virtual camera even if it comes from a real camera on the host pc.

The issue is that you’ll not be able to test the stream injection on any emulator. You’ll always have rejections.

Recommendations

Low delay/motion

The stereo image should have low delay if you’re using aggressive mode. The delay is the time between the 2 frames in the stereo image. We use optical flow to measure the motion between the 2 frames. You may get stereo impaired(1) or stereo impaired(4) warnings if the delay is higher than a certain threshold. These winnings could be ignored if you’re not using the aggressive mode.

High FRR

If you have high FRR (False Rejection Rate) leading to friction, then decrease inject_genuine_threshold.

iOS

Using our online demo at https://www.doubango.org/webapps/face-liveness with an iPhone 13 Pro you’ll see the following list for the available camera:

  1. Front Camera

  2. Back Triple Camera

  3. Back Dual Wide Camera

  4. Back Ultra Wide Camera

  5. Back Dual Camera

  6. Back Camera

  7. Back Telephoto Camera

(1) and (6) work fine. Other cameras intermittently return false-rejection (real camera reported as virtual) which adds friction. We recommend restricting your app to “Front Camera” and “Back Camera” when using it on any iOS device.

Auto-focus

We recommend disabling the auto-focus mode when such feature is supported by the camera. Use the JavaScript function `disableAutoFocus(stream)` provided at https://www.doubango.org/webapps/face-liveness. More about “focusMode” and “exposureMode” at https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints.

Network Authentication Required [7]

Natural images can be provided using drag&drop from any web domain or application. We only accept stereo images (from real or virtual cameras) from “doubango.org” root domain. The network authentication error code 7 means we think “you’re a robot”. Your request may be wrongly rejected (false-rejection). Refresh the browser and try again if you’re not a robot.