> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Configuration settings
> Complete reference for configuring the Cognite PI extractor with YAML settings for authentication, data extraction, backfill, and frontfill operations.
To configure the PI extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org) format. The configuration file is split into [sections](#configure-the-pi-extractor), each represented by a top-level entry in the YAML format. Subsections are nested under a section in the YAML format.
You can use either the sample complete or minimal configuration files included with the [installer](/cdf/integration/guides/extraction/pi/pi_setup) as a starting point for your configuration settings:
* config.default.yml - This file contains all configuration options and descriptions.
* config.minimal.yml - This file contains a minimum configuration and no descriptions.
You must name the configuration file `config.yml`.
You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud.
## Before you start
* Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`.
* The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway.
* Set up an [extraction pipeline](/cdf/integration/guides/interfaces/add_integrations) and note the external ID.
## Minimal YAML configuration file
The YAML settings below contain valid PI extractor 2.1 configurations. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`.
The configuration file has a global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 3 of the configuration schema.
```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
version: 3
cognite:
project: '${COGNITE_PROJECT}'
idp-authentication:
tenant: ${COGNITE_TENANT_ID}
client-id: ${COGNITE_CLIENT_ID}
secret: ${COGNITE_CLIENT_SECRET}
scopes:
- ${COGNITE_SCOPE}
time-series:
external-id-prefix: 'pi:'
pi:
host: ${PI_HOST}
username: ${PI_USER}
password: ${PI_PASSWORD}
state-store:
database: LiteDb
location: 'state.db'
logger:
file:
level: 'information'
path: 'logs/log.log'
```
where:
* `version` is the version of the configuration schema. Use version 3 to be compatible with the Cognite PI extractor 2.1.
* `cognite` is the how the extractor reads the authentication details for PI (pi) and CDF (cognite) from environment variables. Since no host is specified in the `cognite` section, the extractor uses the default value, \<[https://api.cognitedata.com](https://api.cognitedata.com) >, and assumes that the PI server uses Windows authentication.
* `time-series` configures the extractor to create time series in CDF where the external IDs will be prefixed with `pi:`. You can also use a data set ID configuration to add all time series created by the extractor to a particular data set.
* `state-store` configures the extractor to save the extraction state locally using a LiteDB database file named `state.db`.
* `logger` configures the extractor to log at information level and outputs log messages to a log file in the logs/log.log directory. By default, new files are created daily and retained for 31 days. The date is appended to the file name.
## Writing to CDF Data Models
The extractor can write directly to a [core data model time series](/cdf/dm/dm_reference/dm_core_data_model#timeseries), or more correctly, to an extended version of the `CogniteTimeSeries` Data Model type. To save/update instances of `CogniteTimeSeries`, specify the [identifier for the target Data Modeling Space](/api-reference/concepts/20230101/spaces) in the configuration, using the `space-id` parameter of the [time series configuration](#-time-series) section.
The behavior in this mode is almost identical to classic time series destination, except for a few key differences:
* Metadata is written as a json blob to the extended time series type (`CogniteExtractorTimeSeries`), and saved in the `extractedData` property.
* `data-set-id` configuration option is ignored.
## Using values from Azure Key Vault
The PI extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this:
```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
password: !keyvault my-secret-name
```
To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters:
| Parameter | Description |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `keyvault-name` | Name of Key Vault to load secrets from |
| `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. |
| `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. |
| `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. |
| `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. |
**Example:**
```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
key-vault:
keyvault-name: my-keyvault-name
authentication-method: client-secret
tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648
client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6
secret: 1234abcd
```
## Timestamps and intervals
In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places.
For configuring the earliest point to backfill to you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the backfill only recent history.
You can also set the backfill target to a specific date on RFC 3339 form, like `2025-07-02T22:23:12Z`, or on a shorter form like `2022-11-20+12:00`. A timezone specifier is necessary, `Z` means UTC.
## Configure the PI extractor
Configuration for the PI extractor. Each section configures a different aspect of the extractor.
| Parameter | Type | Description |
| ----------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. |
| [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. |
| [`metrics`](#metrics) | object | Configuration for publishing metrics. |
| [`cognite`](#cognite) | object | Configure connection to CDF |
| [`state-store`](#state-store) | object | Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. |
| [`pi`](#pi) | object | Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. |
| [`time-series`](#time-series) | object | Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. |
| [`events`](#events) | object | Configuration for writing events on reconnect and data loss incidents |
| [`extractor`](#extractor) | object | The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:
1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. |
| [`backfill`](#backfill) | object | Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. |
| [`frontfill`](#frontfill) | object | Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. |
| [`high-availability`](#high-availability) | object | Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). |
## `logger`
Global parameter.
Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink.
| Parameter | Type | Description |
| ------------------------------------------ | ------ | -------------------------------------------------------------------------------------------------- |
| [`console`](#logger.console) | object | Configuration for logging to the console. |
| [`file`](#logger.file) | object | Configuration for logging to a rotating log file. |
| [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` |
### `console`
Part of [`logger`](#logger) configuration.
Configuration for logging to the console.
| Parameter | Type | Description |
| -------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. |
| `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. |
### `file`
Part of [`logger`](#logger) configuration.
Configuration for logging to a rotating log file.
| Parameter | Type | Description |
| ------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. |
| `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. |
| `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. |
| `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. |
### `trace-listener`
Part of [`logger`](#logger) configuration.
Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace`
| Parameter | Type | Description |
| --------- | ----------------------------------------------------------------------- | ----------------------------------------------- |
| `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at |
## `metrics`
Global parameter.
Configuration for publishing metrics.
| Parameter | Type | Description |
| ----------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------- |
| [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. |
| [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. |
### `server`
Part of [`metrics`](#metrics) configuration.
Configuration for having the extractor start a Prometheus scrape server on a local port.
| Parameter | Type | Description |
| --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.
**Examples:**
`localhost`
`0.0.0.0` |
| `port` | integer | **Required.** The port used for a local Prometheus server. |
### `push-gateways`
Part of [`metrics`](#metrics) configuration.
A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these.
| Parameter | Type | Description |
| --------------- | ------- | --------------------------------------------------------------------------------------------------- |
| `host` | string | **Required.** URI of the pushgateway host
**Example:**
`http://my.pushgateway:9091` |
| `job` | string | **Required.** Name of the Prometheus pushgateway job. |
| `username` | string | Username for basic authentication |
| `password` | string | Password for basic authentication |
| `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. |
## `cognite`
Global parameter.
Configure connection to CDF
| Parameter | Type | Description |
| ----------------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `project` | string | CDF project to connect to. |
| [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) |
| `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. |
| [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. |
| [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself |
| [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. |
| [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK |
| `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. |
| [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline |
| [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems |
| [`metadata-targets`](#cognite.metadata-targets) | object | Configuration for targets for time series metadata. |
### `idp-authentication`
Part of [`cognite`](#cognite) configuration.
The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow)
| Parameter | Type | Description |
| -------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. |
| `client-id` | string | **Required.** Enter the service principal client id from the IdP. |
| `tenant` | string | Enter the Azure tenant. |
| `token-url` | string | Insert the URL to fetch tokens from. |
| `secret` | string | Enter the service principal client secret from the IdP. |
| `resource` | string | Resource parameter passed along with token requests. |
| `audience` | string | Audience parameter passed along with token requests. |
| `scopes` | configuration for either list or string | |
| `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. |
| [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate |
#### `certificate`
Part of [`idp-authentication`](#cognite.idp-authentication) configuration.
Authenticate with a client certificate
| Parameter | Type | Description |
| --------------- | ------ | ------------------------------------------------------------------------------------------ |
| `authority-url` | string | Authentication authority URL |
| `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication |
| `password` | string | Enter the password for the key file, if it is encrypted. |
### `cdf-retries`
Part of [`cognite`](#cognite) configuration.
Configure automatic retries on requests to CDF.
| Parameter | Type | Description |
| ------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. |
| `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. |
| `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. |
### `cdf-chunking`
Part of [`cognite`](#cognite) configuration.
Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself
| Parameter | Type | Description |
| ------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. |
| `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. |
| `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. |
| `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. |
| `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. |
| `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. |
| `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. |
| `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. |
| `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. |
| `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. |
| `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. |
| `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. |
| `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. |
| `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. |
| `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. |
### `cdf-throttling`
Part of [`cognite`](#cognite) configuration.
Configure the maximum number of parallel requests for different CDF resources.
| Parameter | Type | Description |
| ------------- | ------- | -------------------------------------------------------------------------------------------------- |
| `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. |
| `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. |
| `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. |
| `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. |
| `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. |
| `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. |
| `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. |
| `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. |
### `sdk-logging`
Part of [`cognite`](#cognite) configuration.
Configure logging of requests from the SDK
| Parameter | Type | Description |
| --------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `disable` | boolean | True to disable logging from the SDK, it is enabled by default |
| `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. |
| `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. |
### `extraction-pipeline`
Part of [`cognite`](#cognite) configuration.
Configure an associated extraction pipeline
| Parameter | Type | Description |
| ------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `external-id` | string | External ID of the extraction pipeline |
| `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. |
### `certificates`
Part of [`cognite`](#cognite) configuration.
Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems
| Parameter | Type | Description |
| ------------------------------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks |
| [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates |
#### `allow-list`
Part of [`certificates`](#cognite.certificates) configuration.
List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates
Each element of this list should be a string.
### `metadata-targets`
Part of [`cognite`](#cognite) configuration.
Configuration for targets for time series metadata.
| Parameter | Type | Description |
| ------------------------------------------ | ------ | --------------------------------------------------------- |
| [`raw`](#cognite.metadata-targets.raw) | object | Configuration for writing metadata to CDF Raw. |
| [`clean`](#cognite.metadata-targets.clean) | object | Configuration for enabling writing metadata to CDF Clean. |
#### `raw`
Part of [`metadata-targets`](#cognite.metadata-targets) configuration.
Configuration for writing metadata to CDF Raw.
| Parameter | Type | Description |
| ------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database` | string | **Required.** The Raw database to write to. |
| `timeseries-table` | string | **Required.** Name of the Raw table to write timeseries metadata to, enables writing metadata to Raw. Metadata in this case includes name, description, and unit. |
#### `clean`
Part of [`metadata-targets`](#cognite.metadata-targets) configuration.
Configuration for enabling writing metadata to CDF Clean.
| Parameter | Type | Description |
| ------------ | ------- | ----------------------------------------------------------------------------------- |
| `timeseries` | boolean | Set to `false` to disable writing metadata to time series. Default value is `True`. |
## `state-store`
Global parameter.
Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF.
| Parameter | Type | Description |
| ------------------------ | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. |
| `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. |
| `interval` | string | Enter the time between each write to the state store. 0 or less disables the state store. Format is as given in [Intervals](#intervals). Default value is `10s`. |
| `time-series-table-name` | string | Table name in Raw/Redis/LiteDB for storing time series state. Default value is `ranges`. |
| `extractor-table-name` | string | Table name in Raw/Redis/LiteDB for storing general extractor state. Default value is `extractor`. |
| `redis` | boolean | Use a redis state store. Overrides `database`. |
## `pi`
Global parameter.
Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized.
| Parameter | Type | Description |
| ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host` | string | **Required.** Enter the hostname or IP address of the PI server or the PI collective name. If the host is a collective member name, connect to that member. |
| `username` | string | Enter the username to use for authentication. Leave this empty if the Windows service account under which the extractor runs is authorized in Active Directory to access the PI host. |
| `password` | string | Enter the password for the given username, if any. |
| `native-authentication` | boolean | Determines whether the extractor will use native PI authentication or Windows authentication. The default value is `false`, which indicates Windows authentication. |
| `parallelism` | integer | Insert the number of parallel requests to the PI server. If `backfill-parallelism` is set, this excludes backfill requests. Default value is `1`. |
| `backfill-parallelism` | integer | Insert the number of parallel requests to the PI server for backfills. This allows the separate throttling of backfills. The default value is 0, which means that the value in `parallelism` is used. |
| `use-member-priority` | boolean | When connecting to a PI Collective, attempt to connect to the member with highest priority. If set to `false`, attempt to connect to the member with the same name as `host`. |
| `max-connection-retries` | integer | The maximum number of times to attempt to connect to the PI server before failing fatally. If this is `0`, retry forever. |
## `time-series`
Global parameter.
Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional.
The example would create time series on the following form in CDF:
```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
{
"externalId": "pi:12345",
"name": "PI-Point-Name",
"isString": false,
"isStep": false,
"dataSetId": 1234567890123456
}
```
**Example:**
```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
external-id-prefix: 'pi:'
external-id-source: SourceId
data-set-id: 1234567890123456
```
| Parameter | Type | Description |
| ---------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `external-id-prefix` | string | Enter the external ID prefix to identify the time series in CDF. Leave empty for no prefix. The external ID in CDF will be this prefix followed by either the PI Point name or PI Point ID. |
| `external-id-source` | either `Name` or `SourceId` | Enter the source of the external ID. `Name` means that the PI Point name is used, while `SourceID` means that the PI Point ID is used. Default value is `Name`. |
| `data-set-id` | integer | Specify the data set to assign to all time series controlled by the extractor, both new and current. If you don't configure this, the extractor will not change the current time series' data set. |
| `data-set-external-id` | string | Specify the external ID of the data set to use, see `data-set-id`. Using this requires the dataSets:READ ACL in CDF |
| `sanitation` | either `Remove`, `Clean` or `None` | Specify what to do when the time series fields exceed CDF limits. `Remove` will skip any time series that fail sanitation. `Clean` will truncate and remove values to conform to limits. `None` does nothing (requests may fail as a result). External IDs are never truncated, any time series exceeding CDF limits will be skipped to avoid external ID collisions, regardless of this configuration. Default value is `Clean`. |
| `update-metadata` | boolean | Enable updating time series if extractor configuration or PI metadata changes. Default value is `True`. |
| `space-id` | string | The identifier for the Data modeling space. This extractor configuration parameter overrides `data-set-id` parameter, and changes the extractor destination from the Asset-Centric TimeSeries to a Core Data Model based TimeSeries, using the `CogniteExtractorTimeSeries` system model extension. |
| `source-id` | string | Data modeling source id, this parameter will override the external id of the source node in data modeling, which defaults to the `host` parameter from PI server configuration. |
## `events`
Global parameter.
Configuration for writing events on reconnect and data loss incidents
The example configuration produces events on the following form
```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
{
"externalId": "pi-events:PiExtractor-2020-08-04 01:01:21.395(0)",
"startTime": 1596502850668,
"endTime": 1596502880393,
"type": "DataLoss",
"subtype": "DataPipeOverflow",
"source": "PiExtractor",
"dataSetId": 1234567890123456
}
```
**Example:**
```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
source: PiExtractor
external-id-prefix: 'pi-events:'
data-set-id: 1234567890123456
store-extractor-events-interval: 5m
```
| Parameter | Type | Description |
| --------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source` | string | Events have this value as the `source` in CDF. |
| `external-id-prefix` | string | Set an external ID prefix to identify events in CDF. Leave empty for no prefix. |
| `data-set-id` | integer | Select a data set to assign to all events created by the extractor. We recommend using the same data set ID used in the [Time series](#time-series) section. |
| `data-set-external-id` | string | Data set external ID added to events written to CDF, see `data-set-id`. Using this requires the `dataSets:READ` ACL in CDF |
| `store-extractor-events-interval` | string | Store events in CDF with this interval. Format is as given in [Intervals](#intervals). If this is not set, events are not created in CDF.
**Examples:**
`10s`
`5m` |
## `extractor`
Global parameter.
The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:
1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`.
| Parameter | Type | Description |
| ----------------------------------------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`include-tags`](#extractor.include-tags) | list | Include tags with name exactly equal to one of these names. |
| [`include-prefixes`](#extractor.include-prefixes) | list | Include tags with name starting with one of these prefixes. |
| [`include-patterns`](#extractor.include-patterns) | list | Include tags with name containing one of these substrings. |
| [`include-attribute-values`](#extractor.include-attribute-values) | list | Include tags with attributes equal to at least one of these key/value pairs. |
| [`exclude-tags`](#extractor.exclude-tags) | list | Exclude tags with name exactly equal to one of these names. |
| [`exclude-prefixes`](#extractor.exclude-prefixes) | list | Exclude tags with name starting with one of these prefixes. |
| [`exclude-patterns`](#extractor.exclude-patterns) | list | Exclude tags with name containing one of these substrings. |
| [`exclude-attribute-values`](#extractor.exclude-attribute-values) | list | Exclude tags with attributes equal to at least one of these key/value pairs. |
| `time-series-update-interval` | either string or integer | Interval between checks for new time series in PI. Format is as given in [Intervals](#intervals), this option accepts cron expressions. Default value is `24h`. |
| [`deleted-time-series`](#extractor.deleted-time-series) | object | Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). |
| `end-of-stream-interval` | string | Interval for fetching the end-of-stream timestamps from PI. Format is as given in [Intervals](#intervals). |
| `end-of-stream-chunking` | integer | Maximum number of time series per end-of-stream request. Default value is `10000`. |
| `end-of-stream-throttle` | integer | Maximum number of parallel end-of-stream requests. Default value is `10`. |
| `time-series-update-throttle` | integer | Maximum number of parallel time series updates. Default value is `10`. |
| `dry-run` | boolean | Set this to `true` to run the extractor in dry-run mode, where it does not push any data to CDF. Useful for testing the connection to the PI Server |
| `read-extracted-ranges` | boolean | Set this to `false` to disable reading from CDF on extractor startup. If this is set to `false`, it is strongly recommended to have a state-store configured, or the extractor will read *all* history from PI on every run. Default value is `True`. |
| [`status-codes`](#extractor.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. |
### `include-tags`
Part of [`extractor`](#extractor) configuration.
Include tags with name exactly equal to one of these names.
Each element of this list should be a string.
### `include-prefixes`
Part of [`extractor`](#extractor) configuration.
Include tags with name starting with one of these prefixes.
Each element of this list should be a string.
### `include-patterns`
Part of [`extractor`](#extractor) configuration.
Include tags with name containing one of these substrings.
Each element of this list should be a string.
### `include-attribute-values`
Part of [`extractor`](#extractor) configuration.
Include tags with attributes equal to at least one of these key/value pairs.
| Parameter | Type | Description |
| --------- | ------ | --------------- |
| `key` | string | Attribute name |
| `value` | string | Attribute value |
### `exclude-tags`
Part of [`extractor`](#extractor) configuration.
Exclude tags with name exactly equal to one of these names.
Each element of this list should be a string.
### `exclude-prefixes`
Part of [`extractor`](#extractor) configuration.
Exclude tags with name starting with one of these prefixes.
Each element of this list should be a string.
### `exclude-patterns`
Part of [`extractor`](#extractor) configuration.
Exclude tags with name containing one of these substrings.
Each element of this list should be a string.
### `exclude-attribute-values`
Part of [`extractor`](#extractor) configuration.
Exclude tags with attributes equal to at least one of these key/value pairs.
| Parameter | Type | Description |
| --------- | ------ | --------------- |
| `key` | string | Attribute name |
| `value` | string | Attribute value |
### `deleted-time-series`
Part of [`extractor`](#extractor) configuration.
Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing).
This only affects time series with the same data set ID and external ID prefix as the [time series configured in the extractor](#time-series).
To find time series that exist in CDF but not in PI, the extractor:
* Lists all time series in CDF that have the configured external ID prefix and data set ID.
* Filters the time series using the include/exclude rules defined in the [extractor](#extractor) section.
* Matches the result against the time series obtained from the PI Server after filtering these using the include/exclude rules.
| Parameter | Type | Description |
| ----------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `behavior` | either `None`, `Flag` or `Delete` | Select the action taken by the extractor. Setting this to `Flag` will perform soft deletion: Flag the time series as deleted but don't delete them from CDF. Setting it to `Delete` will delete the time series from CDF.
**If you set this to `Delete`, the time series in CDF that cannot be found in PI will be permanently deleted from CDF.**. Default value is `None`. |
| `flag-name` | string | If you've set `behavior` to `Flag`, use this string to mark the time series as deleted. Metadata is added to the time series with this as the key, and the current time as the value. Default value is `deletedByExtractor`. |
| `time-series-delete-throttle` | integer | Maximum number of parallel delete operations. Default value is `10`. |
### `status-codes`
Part of [`extractor`](#extractor) configuration.
Configuration for ingesting status codes to CDF timeseries.
| Parameter | Type | Description |
| ------------------------ | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. |
## `backfill`
Global parameter.
Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set.
| Parameter | Type | Description |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `skip` | boolean | Set to `true` to disable backfill. |
| `step-size-hours` | integer | Step, in whole number of hours. Set to 0 to freely backfill all time series. Each iteration of backfill will backfill all time series to the next step before stepping further backward. This helps even out the progression of long backfill processes. Default value is `168`. |
| `to` | string | The target CDF timestamp in miliseconds at which to stop the backfill. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.
**Examples:**
`3d-ago`
`2025-07-02T22:23:12Z`
`2022-11-20+12:00` |
| `time-series-delay` | integer | Delay in milliseconds between each time series backfill request within a step. |
| `step-delay` | integer | Delay in milliseconds between each step. |
| `retry-bucket-size` | integer | Maximum size of the retry bucket. Zero or less for no limit. Default value is `100`. |
| `retry-bucket-cleanup-time` | integer | Delay in seconds between each cleanup of the retry bucket. Default value is `3600`. |
## `frontfill`
Global parameter.
Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving.
When data points are archived in PI, they may be subject to compression, reducing the total amount of data points in a time series. Therefore, the backfill and frontfill tasks will receive data points after compression, while the streaming task will receive data points before compression. Learn more about compression in [this video](https://www.youtube.com/watch?v=6-scv3oQ7Kk).
| Parameter | Type | Description |
| -------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `skip` | boolean | Set this to `true` to disable frontfill and streaming. |
| `streaming-interval` | string | Interval between each call to the PI Data Pipe to fetch new data. Format is as given in [Intervals](#intervals). If set to zero, the extractor will not fetch live data from PI.
**If you set this parameter to a high value, there is a higher chance of having a client outbound queue overflow in the PI server. Overflow may cause loss of data points in CDF.**
Recommended range is between 1 and 30 seconds. If you observe data loss events, consider turning this value down. Default value is `1s`. |
| `delete-data-points` | boolean | If you set this to `true`, the corresponding data points are deleted in CDF when data point deletion events are received in the PI Data Pipe.
**Enabling this parameter may increase the streaming latency of the extractor since the extractor verifies the data point deletion before proceeding with the insertions.** |
| `use-data-pipe` | boolean | Older PI servers may not support data pipes. If that's the case, set this value to `false` to disable data pipe streaming. The frontfiller task will run constantly and will frequently query the PI Data Archive for new data points. Default value is `True`. |
| `time-series-chunk` | integer | The maximum number of time series in each frontfill query to PI. The chunks will be adapted according to the density of data points per time series. Default value is `1000`. |
| `data-points-chunk` | integer | The maximum number of requested data points in each frontfill query to PI. Default value is `10000`. |
## `high-availability`
Global parameter.
Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store).
| Parameter | Type | Description |
| ---------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
| `interval` | string | Interval to update the high availability state in Redis. Format is as given in [Intervals](#intervals). |
| `timeout` | integer | Timeout in seconds before taking over as active extractor. Must be set greater than 0 to enable high availability. |