---
title: LAT Transform Configuration
slug: CIOA15BDlRoZXOHNMCJdF
description: Configure data transformation steps in Ocient for customized data processing pipelines.
docTags: 
createdAt: Thu Jul 21 2022 17:05:13 GMT+0000 (Coordinated Universal Time)
updatedAt: Fri Apr 10 2026 20:38:10 GMT+0000 (Coordinated Universal Time)
---

:::hint{type="warning"}
Data Pipelines are now the preferred method for loading data into the Ocient® System. For details, see [Load Data](<./././Load Data.md>).
:::

# `transform`

A transform configuration object.

Required keys:

- Transform Configuration source only: [transform.topics](./#transformtopics)&#x20;
- File sources only: [transform.file\_groups](./#transformtopics)&#x20;

## `transform.topics`

A collection of topics and their associated configuration. Keys are topic names and values are [﻿topic configuration objects](./#topics) . Each `<topic>` set as a key in this object represents an Apache® Kafka® topic defined in Kafka.

:::hint{type="info"}
When loading from an `s3` or `local` File Source type, `transform.file_groups` should be used instead of `transform.topics`. Each key in the `file_group` must match a `file_group_name` defined in the Source section of the pipeline configuration.
:::

| **Type:**     | object |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

## `transform.database`

A database to be used to fully qualify any table names that are not fully qualified.

For example, if `transform.database` is set to `myDatabase`, a table name of the form `schema.table` will become `myDatabase.schema.table`.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

## `transform.schema`

A schema to be used to fully qualify any table names that are not fully qualified. Specifying this property requires that [transform.database](./#transformdatabase)  is specified.

For example, if `transform.database` is set to `myDatabase` and `transform.schema` is set to `mySchema`:

- A table name of the form `table` will become `myDatabase.mySchema.table`
- A table name of the form `schema.table` will become `myDatabase.schema.table`
- A table name of the form `database.schema.table` will stay as `database.schema.table`

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

### Kafka Load Transform Example

```json
{
    ...
    "transform": {
        "topics": {
            "topic_1": {
                "tables": {
                    ...
                }
            },
            "topic_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

### File Based Load Transform Example

Unlike Kafka, File loads define file groups in the source section of the pipeline configuration. The "file\_groups" defined in the source and transform sections must match.

```json
{
    "source": {
        "type": "s3",
        ...
        "file_groups": {
            "file_group_1": {
                ...
            },
            "file_group_2": {
                ...
            },
        }
    }
    "transform": {
        "file_groups": {
            "file_group_1": {
                "tables": {
                    ...
                }
            },
            "file_group_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

## Topics

Topic configuration objects.

Required keys:

- [transform.topics.\<topic>.tables ](./#transformtopicstopictables)&#x20;

:::hint{type="info"}
For file based loads, `topics` are replaced by `file_groups`, but all other settings are equivalent.
:::

### `transform.topics.<topic>.filter`

A record filter to apply at the topic level. See [LAT Record Filtering](<././LAT Data Transformation with JMESPath/LAT Record Filtering.md>) for details.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

### `transform.topics.<topic>.tables`

A collection of tables and their associated configuration. Keys are table names and values are [columns configuration](./#columns).&#x20;

| **Type**:     | object |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

## Tables

Table configuration objects.

Required keys:

- [transform.topics.\<topic>.tables.\<table>.columns](./#transformtopicstopictablestablecolumns)&#x20;

### `transform.topics.<topic>.tables.<table>.filter`

A record filter to apply at the table level. See [LAT Record Filtering](<././LAT Data Transformation with JMESPath/LAT Record Filtering.md>) for details.

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

### `transform.topics.<topic>.tables.<table>.columns`

A collection of columns and their associated configurations. Keys are table names and values are [Columns](./#columns).&#x20;

## Columns

Column transformation configurations.

Required keys:

- [transform.topics.\<topic>.tables.\<table>.columns.\<column> ](./#transformtopicstopictablestablecolumnscolumn)&#x20;

### `transform.topics.<topic>.tables.<table>.columns.<column>`

A column transformation keyed by a column name. A column’s value is defined as a transformation expression. The expression will query the record and return a value that is loaded into the associated column. The grammar of these expressions uses [JMESPath](https://jmespath.org/) enhanced with some custom Ocient transformations and User Defined Transformations (UDTs).

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

# Complex Transform Example

```json
{
    "transform": {
        "database": "adtechdb",
        "schema": "adtech",
        "topics" : {
            "dsp.Auctions" : {
                "tables" : {
                    "auction" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "bidder_nodeid" : "bidder.nodeId",
                            "deals_dealid" : "deals[].dealId"
                        }
                    },
                    "trafficSource" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "sourcesellerid" : "trafficSource.sourceSellerId",
                            "domainname" : "trafficSource.domainName",
                            "contextualdataset_externalentryid" : "trafficSource.siteContextualProfile.contextualDataSet[].entries[].externalEntryId[]",
                            "dimensions_hw" : "join('x', [dimensions.height, dimensions.width])"
                        }
                    },
                    "feedback" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "received" : "received",
                            "campaignid" : "EXPLODE(auctionResponseFeedbacks[].campaignId)",
                            "won" : "EXPLODE(auctionResponseFeedbacks[].won)",
                            "winningbid" : "EXPLODE(auctionResponseFeedbacks[].winningBid)",
                            "sourcelossreasoncode" : "EXPLODE(auctionResponseFeedbacks[].sourceLossReasonCode)"
                        }
                    }
                }
            }
        }
    }
}
```

## Related Links

[Ingest Data with Legacy LAT Reference](<././Ingest Data with Legacy LAT Reference.md>)
