Parallel processing

Parallel processing refers to both mutithreading and batching.

Intra versus Inter parallel processing

We support 2 types of parallel processing: intra and inter.

Intra-processing

The elements processed in parallel are in the same image. For example, when multiple faces are in the same image they’re processed in parallel to extract the embeddings for recognition. Please also note that we heavily use SIMD (Single instruction, multiple data) which is another form of intra parallel processing. The supported extensions are SSE, AVX and NEON. We highly recommend using a CPU with support for AVX2. You’ll have significant speedup even if you don’t have a GPU or NPU.

Inter-processing

The elements processed in parallel are in different images. This mode will be activated if you provide a callback function when initializing the SDK. You’ll have significant speedup only if you have a GPU or NPU. The work will be divided between the GPU/NPU and the CPU.

Multithreading

The number of CPU threads is controlled using configuration entry num_threads. Multithreading works in both inter and intra parallel processing.

Batching

The maximum batch size is controlled using configuration entry max_batchsize. Batching makes sense only if you have a GPU. Batching works in intra parallel processing only.