---
title: LAT Metrics
slug: bszUvgsoSKjbq9jCkzF0i
description: Track and analyze performance metrics in Ocient for comprehensive system monitoring.
docTags: 
createdAt: Thu Jul 21 2022 17:05:13 GMT+0000 (Coordinated Universal Time)
updatedAt: Fri Apr 10 2026 20:40:00 GMT+0000 (Coordinated Universal Time)
---

:::hint{type="warning"}
Data Pipelines are now the preferred method for loading data into the Ocient® System. For details, see [Load Data](<././Load Data.md>).
:::

## Configuration

LAT metrics are exposed using the LAT REST API.

## Endpoints

LAT exposes two metrics endpoints; a ***partitions*** endpoint, and a ***pipeline*** endpoint.

### Partition Metrics

The partitions endpoint exposes individual metrics for every partition that is participating in the current LAT instance. For Apache® Kafka® loads, partitions correspond to Kafka’s partitions. For File Source based loads, partitions correspond to the independent file sets created within each File Group for parallel processing.

The partitions metrics endpoint can be accessed by issuing a curl request as follows:

```curl
curl http://127.0.0.1:8080/v2/metrics/read/lat:type=partitions
```

A response for this request will look as follows, with one entry per partition in the partitions array:

```json
{
    "request": {
        "mbean": "lat:type=partitions",
        "type": "read"
    },
    "value": {
        "partitions": [
            {
                "offsets_durable": 0,
                "pushes_errors": 0,
                "pushes_attempts": 0,
                "rows_pushed": 0,
                "offsets_written": 0,
                "records_buffered": 0,
                "records_errors_column": 0,
                "records_errors_deserialization": 0,
                "records_errors_transformation": 0,
                "offsets_processed": 0,
                "lag": 0,
                "partition": "topic-0",
                "records_filter_accepted": 0,
                "records_errors_row": 0,
                "records_filter_rejected": 0,
                "records_errors_generic": 0,
                "producer_send_attempts": 0,
                "offsets_pushed": 0,
                "pushes_unacknowledged": 0,
                "invalid_state": 0,
                "bytes_pushed": 0,
                "errors_partition": 0,
                "records_errors_total": 0,
                "offsets_buffered": 0,
                "complete": 0,
                "offsets_end": 0,
                "producer_send_errors": 0
            }
        ]
    },
    "timestamp": 1642497992,
    "status": 200
}
```

### Partition Metrics Definitions

Metrics definitions are as follows:

| **Metric**                       | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `partition`                      | Topic partition pair (e.g., `mytopic-0`)                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `records_errors_deserialization` | Total number of processed records which failed to deserialize                                                                                                                                                                                                                                                                                                                                                                                                          |
| `records_errors_transformation`  | Total number of processed records which failed to transform                                                                                                                                                                                                                                                                                                                                                                                                            |
| `records_errors_column`          | Total number of processed records which failed to bind transformed values to columns                                                                                                                                                                                                                                                                                                                                                                                   |
| `records_errors_row`             | Total number of processed records which failed during builder row advancement/other builder errors                                                                                                                                                                                                                                                                                                                                                                     |
| `records_errors_generic`         | Total number of processed records which failed for any other reason                                                                                                                                                                                                                                                                                                                                                                                                    |
| `records_errors_total`           | Sum of all record errors for this topic                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `records_buffered`               | Total number of processed records which have not yet been pushed. Should always be modulo configured buffer size                                                                                                                                                                                                                                                                                                                                                       |
| `records_filter_rejected`        | Number of records rejected by the topic filter.                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `records_filter_accepted`        | Number of records accepted by the topic filter.                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `offsets_processed`              | For Kafka loading, records are processed if their offsets are strictly less than the value of `offsets_processed`.<br /><br />For file loading, this represents the index of the most recently processed file.<br /><br />If there is an error during the processing of a record/file, this metric is still updated according to that record/file’s offset. When a file load is complete, this offset becomes equal to offsets\_end. Can decrease due to reprocessing. |
| `offsets_written`                | For Kafka loading, records are written if their offsets are strictly less than the value of offsets\_written.<br /><br />For file loading, this represents the index of the most recently written file.<br /><br />This metric is only updated when the record/file is processed and the write is successful. Note that a successful write does not mean the record/file is durable. Can decrease due to reprocessing.                                                 |
| `offsets_buffered`               | For Kafka loading, this represents the most recently processed record which was processed to completion and is waiting to be pushed into the Ocient Data Intelligence Platform™.<br /><br />For file loading, this represents the index of the most recently processed file which was processed to completion and is waiting to be pushed into the Ocient data warehouse.                                                                                              |
| `offsets_pushed`                 | For Kafka loading, this represents the highest offset of the batch of rows most recently pushed to the Ocient data warehouse.<br /><br />For file loading, this represents the highest file index of the batch of rows most recently pushed to the Ocient data warehouse.<br /><br />Can decrease due to reprocessing.                                                                                                                                                 |
| `offsets_end`                    | For Kafka loading, this represents the end offset of the partition.<br /><br />For file loading, this represents the file count of the partition.<br />Should never decrease.                                                                                                                                                                                                                                                                                          |
| `offsets_durable`                | For Kafka loading, records are made durable if their offsets are strictly less than the value of offsets\_durable.<br /><br />For file loading, this represents the most durable file index.<br />Should never decrease.                                                                                                                                                                                                                                               |
| `producer_send_attempts`         | Number of Kafka error topic producer send attempts                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `producer_send_errors`           | Number of Kafka error topic producer send errors                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `bytes_pushed`                   | Number of bytes pushed into the Ocient data warehouse                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `rows_pushed`                    | Number of rows pushed into the Ocient data warehouse                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `pushes_attempts`                | Number of attempts to push record batches into the Ocient data warehouse for this partition                                                                                                                                                                                                                                                                                                                                                                            |
| `pushes_errors`                  | Number of attempts to push record batches into the Ocient data warehouse which resulted in error                                                                                                                                                                                                                                                                                                                                                                       |
| `pushes_unacknowledged`          | Number of attempts to push record batches into the Ocient data warehouse for which no response has yet been received                                                                                                                                                                                                                                                                                                                                                   |
| `invalid_state`                  | Number of times a code path was reached in LAT which is erroneous                                                                                                                                                                                                                                                                                                                                                                                                      |
| `complete`                       | Whether or not the partition has any records left to process at the moment; this status can change often in a Kafka load but will likely not change from complete to incomplete in a file load                                                                                                                                                                                                                                                                         |
| `errors_partition`               | The number of times the LAT failed to fetch records for a particular partition                                                                                                                                                                                                                                                                                                                                                                                         |
| `lag`                            | Calculated as offsets\_end - offsets\_durable; In a Kafka load, lag represents the number of unprocessed records. In a file load, lag represents the number of unprocessed files.<br /><br />Lag becomes zero when the complete status is true.                                                                                                                                                                                                                        |



### Pipeline Metrics

The pipeline endpoint exposes aggregate metrics for the LAT instance. There are three categories of pipeline metrics:

| **Pipeline Metric Category** | **Description**                                                                                                                                                                                                                        |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"pipeline" aggregate`       | Monotonically-increasing metrics that act as lifetime counters for the running pipeline. They are independent of the current set of partitions. For instance, `rows_pushed` is the total number of rows ever pushed for this pipeline. |
| `"partitions" aggregate`     | Metrics that depend on the current set of active partitions for the running pipeline. For instance, `total_lag` is the summed lag across all currently active partitions for the pipeline.                                             |
| `pipeline-specific`          | Pipeline only metrics that are not aggregates.                                                                                                                                                                                         |

The Pipeline Metrics endpoint can be accessed by issuing a curl request as follows:

```shell
curl http://127.0.0.1:8080/v2/metrics/read/lat:type=pipeline
```

A response for this request will look as follows:

```json
{
    "request": {
        "mbean": "lat:type=pipeline",
        "type": "read"
    },
    "value": {
        "pipeline": {
            "pushes_errors": 0,
            "pushes_attempts": 0,
            "rows_pushed": 0,
            "records_errors_column": 0,
            "records_errors_deserialization": 0,
            "records_errors_transformation": 0,
            "bytes_pushed": 0,
            "errors_partition": 0,
            "records_filter_accepted": 0,
            "records_errors_total": 0,
            "records_errors_row": 0,
            "records_filter_rejected": 0,
            "records_errors_generic": 0,
            "producer_send_attempts": 0,
            "producer_send_errors": 0
        },
        "partitions": {
            "max_lag": 0,
            "total_offsets_buffered": 0,
            "avg_lag": 0.0,
            "total_pushes_unacknowledged": 0,
            "min_lag": 0,
            "total_lag": 0,
            "total_invalid_state": 0,
            "total_offsets_pushed": 0,
            "total_records_buffered": 0,
            "total_complete": 0,
            "total_offsets_processed": 0
        },
        "paused": 0,
        "bytes_buffered": 0,
        "complete": 0,
        "workers": 32,
        "lat_version": "3.0.0"
    },
    "timestamp": 1642497956,
    "status": 200
}
```

Individual metrics such as `lag`, `offsets_buffered`, `pushes_errors`, etc. are defined within the [Partition Metrics Definitions](./#partition-metrics-definitions) section.

However, metrics specific *only* to the pipeline endpoint are defined as follows:

| **Metric**       | **Description**                                                         |
| ---------------- | ----------------------------------------------------------------------- |
| `paused`         | 0 if processing is active across all workers, 1 otherwise.              |
| `bytes_buffered` | Global allocated memory in bytes.                                       |
| `complete`       | 1 if the pipeline has finished the entire loading process, 0 otherwise. |
| `workers`        | The number of active workers.                                           |
| `lat_version`    | The version of the running LAT server.                                  |

## Related Links

[Ingest Data with Legacy LAT Reference](<./Ingest Data with Legacy LAT Reference.md>)

[System Information REST Endpoints](<././System Administration/Monitoring/System Information REST Endpoints.md>)
