Configuration settings

Extractors that report to Integrations read two configurations. The connection configuration says how to reach CDF and which integration to report to. The application configuration contains extractor-specific settings, as well as common options such as state storage, metrics, and logging.

Integrations is in public preview and may change. This page documents the extractor-side configuration for Integrations. For the API that stores and versions the application configuration, see Configuration.

Connection configuration file

The connection configuration file must be YAML or JSON. Pass it to the extractor with the -c/--connection-config command-line option. All keys use kebab-case.

project: my-project
base-url: https://westeurope-1.cognitedata.com
integration:
  external-id: my-db-extractor
authentication:
  type: client-credentials
  client-id: ${COGNITE_CLIENT_ID}
  client-secret: ${COGNITE_CLIENT_SECRET}
  token-url: https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token
  scopes: https://westeurope-1.cognitedata.com/.default

Base connection object

The following parameters sit at the root of the connection configuration.

Parameter	Type	Description
`project`	string	Required. The CDF project name.
`base-url`	string	Required. The base URL of the CDF cluster, for example `https://westeurope-1.cognitedata.com`.
`integration`	object	Required. Identifies which integration this extractor reports to. See `integration`.
`authentication`	object	Required. Credentials the extractor uses to authenticate with CDF. See `authentication`.
`connection`	object	Network connection settings. See `connection`.

`integration`

This object identifies the integration the extractor reports to.

Parameter	Type	Description
`external-id`	string	Required. The external ID of the integration in CDF that this extractor reports to.

`authentication`

The extractor supports two authentication types, selected with the type key. These parameters are common to both types.

Parameter	Type	Description
`type`	either `client-credentials` or `client-certificate`	Required. Which authentication method to use.
`client-id`	string	Required. The client ID to authenticate with.
`scopes`	string	Required. A space-separated list of OAuth scopes to request, for example `https://westeurope-1.cognitedata.com/.default`.

If type is client-credentials, these parameters apply.

Parameter	Type	Description
`client-secret`	string	Required. The client secret to authenticate with.
`token-url`	string	Required. The URL to fetch tokens from.
`resource`	string	Optional `resource` parameter to pass with token requests.
`audience`	string	Optional `audience` parameter to pass with token requests.

If type is client-certificate, these parameters apply.

Parameter	Type	Description
`path`	string	Required. Path to the client certificate file.
`password`	string	Password for the certificate file, if it is encrypted.
`authority-url`	string	Required. The authentication authority URL.

`connection`

Use these optional settings to control retries and SSL verification for requests to CDF.

Parameter	Type	Description
`retries`	object	Retry behavior for requests to CDF. See `retries`.
`ssl-certificates`	object	SSL verification settings. See `ssl-certificates`.

`retries`

These parameters control how the extractor retries failed requests.

Parameter	Type	Description
`max-retries`	integer	Maximum number of retries for a failed request. Default value is `10`.
`max-backoff`	time interval	Maximum backoff time between retries. See Time interval format. Default value is `30s`.
`timeout`	time interval	Request timeout. See Time interval format. Default value is `30s`.

`ssl-certificates`

These parameters control certificate verification.

Parameter	Type	Description
`verify`	boolean	Whether to verify SSL certificates. Default value is `true`.
`allow-list`	list	List of certificate paths to trust in addition to the system store.

Time interval format

Settings that accept a time interval, such as max-backoff and timeout, use a string with a number followed by a unit: s (seconds), m (minutes), h (hours), or d (days). Examples are 30s and 5m. A plain number without a unit is interpreted as seconds.

Environment variables

Instead of a connection configuration file, an extractor can read its connection settings from environment variables.

Environment variable	Corresponds to
`COGNITE_PROJECT`	`project`
`COGNITE_BASE_URL`	`base-url`
`COGNITE_INTEGRATION`	`integration.external-id`
`COGNITE_CLIENT_ID`	`authentication.client-id`
`COGNITE_TOKEN_SCOPES`	`authentication.scopes`
`COGNITE_CLIENT_SECRET`	`authentication.client-secret` (client-credentials only)
`COGNITE_TOKEN_URL`	`authentication.token-url` (client-credentials only)
`COGNITE_CLIENT_CERTIFICATE_PATH`	`authentication.path` (client-certificate only)
`COGNITE_AUTHORITY_URL`	`authentication.authority-url` (client-certificate only)

Set COGNITE_CLIENT_SECRET to authenticate with client credentials, or COGNITE_CLIENT_CERTIFICATE_PATH to authenticate with a client certificate. The extractor picks the authentication method based on which of these two variables is set.

Application configuration

The application configuration holds settings specific to what the extractor does, such as which queries or tags to run. Every extractor that reports to Integrations also supports the following common settings.

Parameter	Type	Description
`state-store`	object	Where the extractor persists its state between runs. See `state-store`.
`metrics`	object	Where the extractor sends performance metrics. See `metrics`.
`log-handlers`	list	Where the extractor writes logs. See `log-handlers`. Default is a single console handler at the `info` level.

By default, the extractor fetches the active revision of the application configuration from the integration’s configuration revisions. Use -f/--force-local-config to load the application configuration from a local file instead. See Command line options.

`state-store`

This object controls where the extractor persists its state between runs.

Parameter	Type	Description
`type`	either `raw` or `local`	Required. Where to store extractor state.

If type is raw, these parameters apply.

Parameter	Type	Description
`database`	string	Required. The CDF RAW database to store state in.
`table`	string	Required. The RAW table to store state in.
`upload-interval`	time interval	How often to save state to RAW. Default value is `30s`.

If type is local, these parameters apply.

Parameter	Type	Description
`path`	string	Required. Path to a local JSON file to store state in.
`save-interval`	time interval	How often to save state to the file. Default value is `30s`.

`metrics`

You can send metrics to a Prometheus push gateway, to CDF as time series, or to a local Prometheus endpoint.

Parameter	Type	Description
`push-gateways`	list	One or more Prometheus push gateways to send metrics to. See `push-gateways`.
`cognite`	object	Push metrics as CDF time series. See `cognite`.
`server`	object	Expose metrics on a local Prometheus endpoint. See `server`.

`push-gateways`

Each list item is a Prometheus push gateway.

Parameter	Type	Description
`host`	string	Required. URL of the push gateway.
`job-name`	string	Required. Job name to push metrics under.
`username`	string	Username for the push gateway, if it requires authentication.
`password`	string	Password for the push gateway, if it requires authentication.
`push-interval`	time interval	How often to push metrics. Default value is `30s`.
`clear-after`	time interval	If set, clear the gateway this long after the extractor stops.

`cognite`

These parameters control metrics stored as CDF time series.

Parameter	Type	Description
`external-id-prefix`	string	Required. Prefix used on the external IDs of the time series the extractor creates.
`asset-name`	string	Name of an asset to attach the metric time series to.
`asset-external-id`	string	External ID of an asset to attach the metric time series to.
`data-set`	object	Data set (by `id` or `external-id`) to put the metric time series in.
`push-interval`	time interval	How often to push metrics. Default value is `30s`.

`server`

These parameters expose a local Prometheus endpoint.

Parameter	Type	Description
`port`	integer	Port to expose the Prometheus endpoint on. Default value is `9000`.
`host`	string	Host to bind the Prometheus endpoint to. Default value is `0.0.0.0`.

`log-handlers`

log-handlers is a list of one or more log handlers. Select each handler with the type key. If type is console, these parameters apply.

Parameter	Type	Description
`level`	one of `critical`, `error`, `warning`, `info`, `debug`	Required. Minimum level to log to standard output.

If type is file, these parameters apply.

Parameter	Type	Description
`path`	string	Required. Path to the log file.
`level`	one of `critical`, `error`, `warning`, `info`, `debug`	Required. Minimum level to log to the file.
`retention`	integer	Number of daily log files to keep. Default value is `7`.

Command line options

These options apply to extractors that report to Integrations.

Option	Description
`-c`, `--connection-config`	Required. Path to the connection configuration file.
`-f`, `--force-local-config`	Path to a local application configuration file, used instead of fetching the configuration revision from CDF.
`-l`, `--log-level`	Overrides the console log level from `log-handlers` for this run. One of `debug`, `info`, `warning`, `error`, `critical`.
`--skip-init-checks`	Skips startup checks. Useful for debugging. Do not use in production.
`--cwd`	Sets the working directory for the extractor.
`--service`	Runs the extractor as a Windows service. Only supported on Windows.
`--bootstrap-log-file`	Path to a log file that captures startup logs before the application configuration loads. Default value is `logs/bootstrap.log`.

Data engineering

Connection configuration file

Base connection object

`integration`

`authentication`

`connection`

`retries`

`ssl-certificates`

Time interval format

Environment variables

Application configuration

`state-store`

`metrics`

`push-gateways`

`cognite`

`server`

`log-handlers`

Command line options

Further reading

​Connection configuration file

​Base connection object

​integration

​authentication

​connection

​retries

​ssl-certificates

​Time interval format

​Environment variables

​Application configuration

​state-store

​metrics

​push-gateways

​cognite

​server

​log-handlers

​Command line options

​Further reading

Connection configuration file

Base connection object

`integration`

`authentication`

`connection`

`retries`

`ssl-certificates`

Time interval format

Environment variables

Application configuration

`state-store`

`metrics`

`push-gateways`

`cognite`

`server`

`log-handlers`

Command line options

Further reading