To create a detector unit the developer have to write the
nvinfer@detector declaration like it’s shown in the following listing:
- element: nvinfer@detector name: Primary_Detector model: format: caffe model_file: resnet10.caffemodel batch_size: 1 precision: int8 int8_calib_file: cal_trt.bin label_file: labels.txt input: scale_factor: 0.0039215697906911373 output: num_detected_classes: 4 layer_names: [conv2d_bbox, conv2d_cov/Sigmoid]
Full documentation for the detector unit is located on the unit page:
You must configure model parameters, input and output. The parameters also may be specified in the model.config_file file according to Nvidia DeepStream specification for a detector (example). YAML-based parameters override those configured in model.config_file.
The batch size is directly connected with inference performance. When handling multiple streams consider setting
batch_size equal to maximum expected number of streams. For Jetson family Nvidia often uses the batch size equal to
32 in their benchmarks.
The batching is a sophisticated topic: there is no silver bullet for selecting a batch size, especially when inference happens on the second and later steps; experiment with your values to achieve the best possible performance and GPU utilization.
input block describes how the unit preprocesses the data and selects objects which are the subject of operation. All the
input parameters can be found in the specification for
Input defines several parameters which are important to understand. However, here we would like to highlight the
object parameter which specifies a label for objects selected for inference. If the
object parameter is not set, it is assumed to be equal to
frame: default object with ROI covering the whole frame (without paddings added).
The primary detectors usually work with the default
object: frame, while secondary detectors specify labels to select objects of interest.
There are other parameters in the
input section which select objects, like
object_min_height and others. Please, keep in mind that the selected objects are passed to the model, and those not selected are ignored, but they continue to persist in metadata, so downstream elements can consume them later.
Imagine the situation when you have 2 models: the first one works well with medium-size objects but is very heavyweight while the second works perfectly with large objects and is lightweight. To efficiently utilize resources you may place two inference elements with properly defined input blocks:
- element: nvinfer@detector name: Secondary_Detector_Small model: format: caffe model_file: model1 input: object_min_height: 30 object_max_height: 100 ... - element: nvinfer@detector name: Secondary_Detector_Large model: format: caffe model_file: model2 input: object_min_height: 101
Input also allows making custom preprocessing with
preprocess_object_tensor. To use the former, you have to implement code with the interface
BasePreprocessObjectMeta, the later is an advanced topic which is not covered here. Keep in mind, that all modifications made in preprocess are restored after the unit is done with its processing.
Example of preprocess_object_meta:
input: object: object_detector.something preprocess_object_meta: module: something_detector.input_preproc class_name: TopCrop
"""Model input metadata preprocessing: cropping variations.""" from savant_rs.primitives.geometry import BBox from savant.base.input_preproc import BasePreprocessObjectMeta from savant.meta.object import ObjectMeta class CropTopPreprocessObjectMeta(BasePreprocessObjectMeta): """Make bbox height no lesser that bbox width.""" def __call__(self, object_meta: ObjectMeta) -> BBox: bbox = object_meta.bbox max_dim = max(bbox.width, bbox.height) return BBox( bbox.xc, bbox.top + max_dim / 2, bbox.width, max_dim, )
output section describes how the unit processes metadata before passing them to the following unit. The parameters of
output may be found in the specification for
layer_names defines the names of the model output layers returned for postprocessing.
Output defines an important parameter
converter which is basically a method which makes bounding boxes from a raw tensor. For “standard” detection models supported by DeepStream
converter parameter is not required, however if the model’s output cannot be parsed automatically, you have to provide an implementation of
BaseObjectModelOutputConverter to produce boxes for detected objects.
converter: module: savant.converter.yolo_x class_name: TensorToBBoxConverter kwargs: decode: true
The converter implementation can be found in the class
An example of the converter for YOLOv4 listed below. The YOLOv4 model has two output layers: the first represents box definition (incl.
class_id), the last is for confidence. When you are writing the converter you must return objects relative to the ROI of the parent object.
class TensorToBBoxConverter(BaseObjectModelOutputConverter): """`YOLOv4 <https://github.com/Tianxiaomo/pytorch-YOLOv4>`_ output to bbox converter.""" def __call__( self, *output_layers: np.ndarray, model: ObjectModel, roi: Tuple[float, float, float, float], ) -> np.ndarray: """Converts detector output layer tensor to bbox tensor. :param output_layers: Output layer tensor :param model: Model definition, required parameters: input tensor shape, maintain_aspect_ratio :param roi: [top, left, width, height] of the rectangle on which the model infers :return: BBox tensor (class_id, confidence, xc, yc, width, height, [angle]) offset by roi upper left and scaled by roi width and height """ boxes, confs = output_layers roi_left, roi_top, roi_width, roi_height = roi # [num, 1, 4] -> [num, 4] bboxes = np.squeeze(boxes) # left, top, right, bottom => xc, yc, width, height bboxes[:, 2] -= bboxes[:, 0] bboxes[:, 3] -= bboxes[:, 1] bboxes[:, 0] += bboxes[:, 2] / 2 bboxes[:, 1] += bboxes[:, 3] / 2 # scale if model.input.maintain_aspect_ratio: bboxes *= min(roi_width, roi_height) else: bboxes[:, [0, 2]] *= roi_width bboxes[:, [1, 3]] *= roi_height # correct xc, yc bboxes[:, 0] += roi_left bboxes[:, 1] += roi_top # [num, num_classes] --> [num] confidences = np.max(confs, axis=-1) class_ids = np.argmax(confs, axis=-1) return np.concatenate( ( class_ids.reshape(-1, 1).astype(np.float32), confidences.reshape(-1, 1), bboxes, ), axis=1, )
output you may also select only necessary objects by specifying their IDs and labels:
output: layer_names: [output_bbox/BiasAdd, output_cov/Sigmoid] num_detected_classes: 3 objects: - class_id: 0 label: person selector: kwargs: min_width: 32 min_height: 32 - class_id: 2 label: face selector: kwargs: confidence_threshold: 0.1
All skipped classes will be permanently excluded from the next steps of the pipeline. The
selector block also allows defining a filter to eliminate unnecessary objects.
If unit name is
Primary_Detector, then to address selected objects in the following units use
The default selector implementation runs NMS and allows selecting objects by specifying
confidence_threshold. To create a custom
selector you have to implement
BaseSelector. You may take a look at
BBoxSelector to get an idea of how to write it.