Prometheus Metrics Support

Savant supports Prometheus metrics of two kinds:

system-level metrics collected by the pipeline and adapters and provided as is;
user-level metrics managed by the user.

The metrics are exported in the Open Metrics textual format (currently known as a Prometheus format).

The system-level metrics mostly measure queues, frames, detected objects, latencies and other valuable information related to pipeline operation.

The user-level metrics must reflect user needs.

There are two samples demonstrating metrics:

The Buffer Adapter sample: demonstrates buffer-collected metrics;
The Pass-through Processing sample: demonstrates adapter system-level metrics and user-level metrics.

The system-level metrics are collected according to the collection periods, configured for the pipeline with the following parameters:

parameters:
  telemetry::
    # Configure metrics
    metrics:
      # Output stats after every N frames
      frame_period: ${oc.decode:${oc.env:METRICS_FRAME_PERIOD, 10000}}
      # Output stats after every N seconds
      time_period: ${oc.decode:${oc.env:METRICS_TIME_PERIOD, null}}
      # How many last stats to keep in the memory
      history: ${oc.decode:${oc.env:METRICS_HISTORY, 100}}
      # Parameters for metrics provider
      extra_labels: ${json:${oc.env:METRICS_EXTRA_LABELS, null}}

Configuration parameters for the system-level metrics collection:

Parameters
Parameter	Environment variable	Description	Default	Example
`parameters.telemetry.metrics.frame_period`	`METRICS_FRAME_PERIOD`	The number of frames after which the metrics are collected.	`10000`	`1000`
`parameters.telemetry.metrics.time_period`	`METRICS_TIME_PERIOD`	The number of seconds after which the metrics are collected.	`null`	`10`
`parameters.telemetry.metrics.history`	`METRICS_HISTORY`	The number of last stats to keep in memory. This value is required only for fetching the history with `savant-rs` from the pipeline runtime.	`100`	`10`
`parameters.telemetry.metrics.extra_labels`	`METRICS_EXTRA_LABELS`	Additional labels for the metrics which are automatically added to every metric (both user- and system-level).	`null`	`{"label1": "value1", "label2": "value2"}`

User-Level Metrics

User-level metrics are managed by the user and configured in PyFunc blocks with the savant-rs interface.

Savant supports two metric types:

CounterFamily (64-bit unsigned int);
GaugeFamily (64-bit float).

The metrics usage is demonstrated in the Pass-through Processing sample. Those two functions are enough in most cases; they use simplified wrappers implemented in Savant for user convenience.

Complete Metric API can be found in the savant-rs documentation.