Configure a simulator connector
The features described in this section are currently in beta testing and are subject to change.
Simulator connectors can be downloaded from Data management > Integrate > Extractors in Cognite Data Fusion (CDF). Then search for a connector by name; for example PROSPER.
To configure a simulator connector, you must edit the configuration file. The file is in YAML format, and the sample configuration file contains all valid options with default values. You can exclude fields entirely to let the connector use default values. The configuration file separates settings by component, and you can remove an entire component to disable it or use the default values.
You'll find configuration file examples listed under the config
folder in the directory where the connector is installed.
Don't base your configuration on the config.example.yml
file. Instead, use config.minimal.yml
as your base and copy the parts you need from config.example.yml
.
Using values from environment variables
In the configuration file, values wrapped in ${}
are replaced with environment variables with that name. For example, ${COGNITE_PROJECT}
will be replaced with the value of the environment variable called COGNITE_PROJECT
.
The configuration file also contains the global parameter version
, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema.
You can set up extraction pipelines to use versioned connector configuration files stored in the cloud.
Connector
This section contains information on how the current connector appears in Simulators.
Parameter | Description |
---|---|
connector.name-prefix | Enter the connector name that is registered on the API. |
connector.add-machine-name-suffix | Set to true to have the connector name appear as name@vm-host-name . |
connector.api-logger.enabled | Insert boolean of whether to push logs to CDF or not. The default value is false . |
connector.api-logger.level | Enter the minimum level of logs to push to CDF. Valid values are Information , Debug , Error , and Warning . |
Logger
Log entries are either Fatal
, Error
, Warning
, Information
, Debug
, Verbose
, in order of decreasing importance. Each level covers the ones of higher importance.
Parameter | Description |
---|---|
console | Configuration for logging to the console. |
console.level | Set the minimum level of log events to write to the console. Set this parameter to enable console logging. |
console.stderr-level | Log events at this level or above are redirected to standard error. |
file | Configuration for logging to a rotating log file. |
file.level | Set the minimum level of log events to write to file. |
file.path | Enter the path to the files to be logged. For instance, if this is set to logs/log.txt , logs on the form logs/log[date].txt will be created, depending on rolling-interval . |
file.retention-limit | Set the maximum number of log files that are kept in the log folder. |
file.rolling-interval | Enter a rolling interval for log files as day or hour . The default value is day . |
Cognite
This is the configuration of the connection to CDF.
Parameter | Description |
---|---|
host | Insert the CDF service URL. The default value is https://api.cognitedata.com . |
project | Enter the CDF project. This is a required parameter. |
idp-authentication | The configuration for authenticating to CDF. |
idp-authentication.authority | The authority used with tenant to authenticate to Azure tenants. Use token-url if you're connecting using a non-azure IdP. The default value is https://login.microsoftonline.com . |
idp-authentication.tenant | Enter the Azure tenant used with authority . |
idp-authentication.token-url | Insert the URL used to obtain service tokens, used for non-azure IdPs. |
idp-authentication.client-id | Enter the service principal client ID. |
idp-authentication.secret | Enter the service principal client secret. |
idp-authentication.resource | Insert an optional resource parameter to pass along with token requests. |
idp-authentication.scopes | Enter a list of scopes to pass along with the request. This will typically need to contain [host]/.default . |
idp-authentication.audience | Insert an optional audience parameter to pass along with token requests. |
idp-authentication.min-ttl | Set the requested minimum time-to-live in seconds for the token. |
idp-authentication.certificate | Configuration for authenticating using a client certificate. |
idp-authentication.certificate.authority-url | Insert the certificate authority URL. |
idp-authentication.certificate.path | Enter the path to the .pem or .pfx certificate to be used for authentication. |
idp-authentication.certificate.password | Enter the certificate password. |
max-upload-interval | Set the maximum time to cache data points before they are uploaded to CDF. The syntax is described in Timestamps and intervals. The default value is 1s . |
max-data-points-upload-queue-size | Set the maximum number of cached data points before they are uploaded to CDF. The default value is 1000000 . |
cdf-retries | Set the number of automatic retries on requests to CDF. |
cdf-retries.timeout | Set a timeout in milliseconds for each individual try. The default value is 80000 . |
cdf-retries.max-retries | Set the maximum number of retries. A value less than 0 retries forever. |
cdf-retries.max-delay | Set the maximum delay between each try in milliseconds. The base delay is calculated according to 125 * 2 ^ retry milliseconds. If this is less than 0, there's no upper limit. The default value is 5000 . |
cdf-chunking | The configuration for chunking on requests to CDF. Note that increasing these may cause requests to fail, due to limits in the API. Read the API documentation before making these higher than their current value. |
cdf-chunking.time-series | Set the maximum number of time series per get/create time series request. |
cdf-chunking.assets | Set the maximum number of assets per get/create assets request. |
cdf-chunking.data-point-time-series | Set the maximum number of time series per data point create request. |
cdf-chunking.data-points | Set the maximum number of data points per data point create request. |
cdf-throttling | Set how requests to CDF should be throttled. |
cdf-throttling.time-series | Set the maximum number of parallel requests per time series operation. The default value is 20 . |
cdf-throttling.assets | Set the maximum number of parallel requests per assets operation. The default value is 20 . |
cdf-throttling.data-points | Set the maximum number of parallel requests per datapoints operation. The default value is 10 . |
sdk-logging | The configuration for logging of requests from the SDK. |
sdk-logging.disable | Set this to true to disable logging of requests from the SDK. This is by default enabled. |
sdk-logging.level | Set the log level to log messages from the SDK. The default value is debug . |
sdk-logging.format | Set the log message format. The default value is CDF ({Message}): {HttpMethod} {Url} - {Elapsed} ms |
nan-replacement | Replacement for NaN values when writing to CDF. The default value is none, meaning these values are removed. |
extraction-pipeline.external-id | The configuration for associating this connector with an extraction pipeline. This is used for monitoring and remote configuration. |
certificates | The configuration for special handling of SSL certificates. This should not be considered as a permanent solution to certificate problems. |
certificates.accept-all | Accept all remote SSL certificates even if verification fails. Note: This introduces a risk of man-in-the-middle attacks. |
certificates.allow-list | List of certificate thumbprints to automatically accept. This is a smaller risk than accepting all certificates. |
State store
This section includes the configuration parameters for storing state in a local database or in CDF RAW.
Parameter | Description |
---|---|
location | Enter the path to the database file, or name of RAW database containing state store. |
database | Select which type of database to use. Valid options are LiteDb or None . The default value is None . |
interval | Set the interval between each push of local states to the state store. The syntax is described in Timestamps and intervals. The default value is 1m . |