Improving the memory usage¶
The SDK is very light and will use very few Mb of your RAM. You can use the Benchmark sample app to measure memory usage.
- For example, on Raspberry Pi 3+ with 1GB RAM:
8% for sequential processing
17% for parallel processing
Danger
Please note that we use memory pooling to allocate space in advance. You may see memory usage grow before any processing.
Reduce the number of auxiliary engines¶
We use auxiliary inference engines in parallel to improve the frame rate. This option is different than parallel processing feature and always enabled.
You can change the number of auxiliary engine by passing the info to the plugin (inference engine) using json configuration entry plugin_options. The entry accepts key-value-pairs separated by semicolon (e.g. "num_aux:2;batching:y;num_threads:8").
To set the number of auxiliaries to zero, the json entry would look like this:
"plugin_options": "num_aux:0"
GPU memory shrinkage and workspace limitation¶
ONNX RT keeps incrementing allocated GPU memory each time the batch size increases. You can change that and instruct the engine to free the extra memory after inference and limit the amount to the model’s weights and parameters. You can also limit the amount of memory allocated for the GPU workspace.
To do the above restrictions, set the plugin_options with memory.enable_memory_arena_shrinkage:gpu$cpu;cudnn_conv_use_max_workspace:0;session.use_device_allocator_for_initializers:1;. Check Microsoft documentation for more information.
Use another processing unit (GPU, NPU…)¶
Using another processing unit (GPU, NPU…) for inference will reduce memory/power usage for the CPU. In such case, the CPU will be used only for image pre-processing which will result in few Kbs usage instead of few Mbs.
Use QINT8 models instead of FLOAT32¶
All models (TfLite, OpenVINO, Hailo, RKNPU2…) are quantized (QINT8) except ONNX (FLOAT32) and TensorRT (FLOAT16). QINT8 models are up to #4 times smaller with marginal accuracy loss thanks to Quantization aware training (QAT).
Use Sequential processing¶
Parallel processing could be up to #5 times faster than sequential mode and that comes with a cost, higher memory usage. Parallel processing could consume up to 1.5 times more memory.
Danger
Disabling parallel processing will significantly slow down the processing on some devices.
Disable number
Disable batching¶
Let’s say an image contains #5 license plates, then the SDK will call the inference function with batch size #5 which means they will be processed in parallel. Obviously, processing #5 images in parallel will consume #5 times more memory than processing them #1 by #1.
Set batching_mode json configuration entry to off to disable it.
Danger
Disabling batching may significantly slow down the processing specially if you’re using a GPU.