---
title: Configuration Settings for Data Pipelines
slug: cU5o-configur
docTags: 
createdAt: Wed Feb 19 2025 19:38:35 GMT+0000 (Coordinated Universal Time)
updatedAt: Mon Mar 30 2026 18:52:37 GMT+0000 (Coordinated Universal Time)
---

Data pipeline functionality enables the use of SQL statements to load data. You can use a variety of configuration settings to manage the load. See this table for the settings that you can use.

| **Configuration Setting Parameter Name**                                                                      | **Default**                                                                                                          | **Data Type** | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------- |   |
| `streamloader.extractorEngineParameters.enabled`                                                              | `true`                                                                                                               | BOOLEAN       | Whether the Loader Node attempts to bootstrap the extractor Java® process for data pipelines on start-up. If you set this parameter to `true`, the Loader Node does not become active until the Java loading process has successfully started.                                                                                                                                                                                                                                                                                                 |
| `streamloader.extractorEngineParameters.jvmMemoryConfiguration`                                               | - `32g` for `initialHeap`
- `64g` for `maxHeap`
- `64g` for `maxDirect`                                              | VARCHAR       | The JVM®  memory configuration to use for the extractor Java process. Use strings representing data sizes in SI format: 1 KiB, 5 MiB, etc.<br />You must set the three memory options in the same statement. <br /><br />**Example**<br />`ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.jvmMemoryConfiguration.initialHeap' = '32g',`&#xA;`'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxHeap' = '64g',`&#xA;`'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxDirect' = '64g';` |
| `streamloader.extractorEngineParameters.restPort`                                                             | 8090                                                                                                                 | INT           | Use this parameter with the `streamloader.extractorEngineParameters.portOffset` parameter to form the REST port for the extractor. (External monitoring services like Telegraf™ can query the REST server for loading process metrics.)                                                                                                                                                                                                                                                                                                        |
| `streamloader.extractorEngineParameters.portOffset`                                                           | 0                                                                                                                    | INT           | Use this parameter with the `streamloader.extractorEngineParameters.restPort` parameter to form the REST port of the extractor. (External monitoring services like Telegraf can query the REST server for loading process metrics.)                                                                                                                                                                                                                                                                                                            |
| `streamloader.extractorEngineParameters.configurationOption.expect.empty.file.list`                           | `false`                                                                                                              | BOOLEAN       | Set this value to `false` to fail a file source data pipeline that finds no files to load instead of the load completing successfully with no data loaded.                                                                                                                                                                                                                                                                                                                                                                                     |
| `streamloader.extractorEngineParameters.configurationOption.pipeline.preview.rows.limit`                      | 1000                                                                                                                 | INT           | The maximum number of rows a `PREVIEW PIPELINE` SQL statement can return. For example, when you use this statement, if the specified `LIMIT` value exceeds this number, then the loading process throws an error.                                                                                                                                                                                                                                                                                                                              |
| `streamloader.extractorEngineParameters.configurationOption.engine.transform.udt.jarRootDirectory`            | `/opt/ocient/current/lib/extractorengine_udt`                                                                        | VARCHAR       | The absolute path to the directory containing JARs for data pipeline functions. To install and enable a third-party library for using data pipeline functions, install the JAR package at this location on all Loader Nodes. Then, add the chosen fully-qualified class name to the IMPORT clause of the data pipeline function. For details, see [CREATE OR REPLACE PIPELINE FUNCTION](<./././SQL Reference/Data Definition Language _DDL_ Statement Reference/Data Pipelines.md>).                                                           |
| `streamloader.extractorEngineParameters.configurationOption.engine.external.jdbc.jarRootDirectory`            | `/opt/ocient/current/lib/extractorengine_jdbc`                                                                       | VARCHAR       | The absolute path to the directory containing JARs for external source lookup functionality. For details, see [Load Data from External Sources in Data Pipelines](<./././Load Data/Load Data from External Sources in Data Pipelines.md>).                                                                                                                                                                                                                                                                                                     |
| `streamloader.extractorEngineParameters.configurationOption.source.record.max.size`                           | Dynamically calculated value at start-up based on the number of processors and amount of memory available to the JVM | BIGINT        | The maximum source record size, in bytes, that the loading process tolerates before throwing an error.                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `streamloader.extractorEngineParameters.configurationOption.s3.region`                                        | `us-east-1`                                                                                                          | VARCHAR       | The default region for Amazon® Web Services℠ (AWS℠) S3 file sources.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access`                       | `false`                                                                                                              | BOOLEAN       | Set this value to `true` to use path-style access for all S3 file sources. The default is a virtual-hosted style.                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access.if.endpoint.overriden` | `true`                                                                                                               | BOOLEAN       | Set this value to `true` to use path-style access for S3 file sources that specify endpoints. (The system still uses virtual-hosted style access for sources without endpoint overrides unless you set `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access` to `true`).                                                                                                                                                                                                                                     |
| `streamloader.extractorEngineParameters.configurationOption.s3.retries.count`                                 | 10                                                                                                                   | INT           | The number of retries for the S3 file source client.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `streamloader.extractorEngineParameters.configurationOption.s3.netty.read.timeout.seconds`                    | 0                                                                                                                    | INT           | The timeout value for read operations, in seconds, for the S3 file source client. `0` means do not perform a timeout.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `streamloader.extractorEngineParameters.configurationOption.s3.netty.max.concurrency`                         | 50                                                                                                                   | INT           | Maximum number of allowed concurrent requests for the S3 file source client.                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.base.delay.seconds`       | 1                                                                                                                    | INT           | The base amount of time, in seconds, for calculating the time the S3 file source client waits before retrying a failed request. The range of values that the calculated time can achieve increases exponentially with each failure. You can set the maximum value using the `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds` parameter.                                                                                                                                                |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds`      | 20                                                                                                                   | INT           | The maximum amount of time the S3 file source client waits before retrying a failed request.                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.max.pending.connection.acquires`           | 20000                                                                                                                | INT           | The maximum number of pending connection acquires that the S3 file source client allows.                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.timeout.seconds`                | 10                                                                                                                   | INT           | The amount of time, in seconds, that the S3 file source client waits when initially establishing a connection.                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.max.idle.timeout.seconds`       | 300                                                                                                                  | INT           | The maximum amount of time, in seconds, that the system allows a connection established by the S3 file source client to remain open while idle.                                                                                                                                                                                                                                                                                                                                                                                                |
| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.acquisition.timeout.seconds`    | 300                                                                                                                  | INT           | The amount of time, in seconds, that the S3 file source client waits when acquiring a connection from the pool.                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `streamloader.extractorEngineParameters.configurationOption.kafka.poll.timeout.msec`                          | 1000                                                                                                                 | BIGINT        | The amount of time, in milliseconds, that a consumer for a pipeline with an Apache® Kafka® source waits for records to become available during a poll operation.                                                                                                                                                                                                                                                                                                                                                                               |
| `streamloader.extractorEngineParameters.configurationOption.kafka.assign.timeout.msec`                        | 20000                                                                                                                | BIGINT        | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for partitions to be assigned to the consumer on a pipeline with a Kafka source. If the system does not assign any partitions when the timeout elapses, then the `PREVIEW PIPELINE` statement terminates without data processing.                                                                                                                                                                                                                           |
| `streamloader.extractorEngineParameters.configurationOption.kafka.activate.timeout.msec`                      | 5000                                                                                                                 | BIGINT        | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for a partition to receive records on a pipeline with a Kafka source.                                                                                                                                                                                                                                                                                                                                                                                       |
| `streamloader.extractorEngineParameters.configurationOption.arrow.max.native.memory.usage.bytes`              | 23756537856                                                                                                          | BIGINT        | The default maximum native memory usage, in bytes, for the Apache® Parquet™ reader.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `streamloader.extractorEngineParameters.configurationOption.arrow.max.java.buffer.memory.usage.bytes`         | 15837691904                                                                                                          | BIGINT        | The default maximum buffer memory usage, in bytes, for the Parquet reader.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `streamloader.extractorEngineParameters.configurationOption.filesystem.access.directories`                    | `/tmp`                                                                                                               | VARCHAR       | Comma-separated list of directories where you can load server file system data.<br /><br />Each directory must exist.<br /><br />All files must be located within one of the directories.<br /><br />By default, you can load from the `/tmp` directory. <br />                                                                                                                                                                                                                                                                                |

# Related Links

[Load Data](<./././Load Data.md>)

[Alter Default Data Pipeline Behavior](<./Alter Default Data Pipeline Behavior.md>)&#x20;
