Configuration settings - Cognite Docs

To configure the PI replace utility, you must create a configuration file. The file must be in YAML format. In addition to the sections listed below, you must include these PI extractor configuration sections:

Minimal YAML configuration file

The YAML settings below contain valid PI replace utility configurations, but not all configuration options are used. The values wrapped in ${} are replaced with environment variables with that name. For example, ${COGNITE_PROJECT} will be replaced with the value of the environment variable called COGNITE_PROJECT. The configuration file also contains the global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 2 of the configuration schema.

version: 2

cognite:
    project: ${COGNITE_PROJECT}
    idp-authentication:
        tenant: ${COGNITE_TENANT_ID}
        client-id: ${COGNITE_CLIENT_ID}
        secret: ${COGNITE_CLIENT_SECRET}
        scopes:
          - ${COGNITE_SCOPE}

time-series:
    external-id-prefix: "pi:"

pi:
    host: ${PI_HOST}
    username: ${PI_USER}
    password: ${PI_PASS}

metrics:
    push-gateways:
        - host: "http://localhost:9091"
          job: "pi-replace-job"

state-store:
    database: LiteDb
    location: "PiReplaceStateStore.db"

logger:
    console:
        level: "information"

replace:
    start-time: "2020-02-20 10:05:00"
    end-time: "2020-03-20 10:05:00"
    save-progress-interval: 10s
    status-codes:
        status-codes-to-ingest: All

data-quality:
    check-data-quality: true
    report-file-name: "report"
    ignore-positive-difference: true

Configure the PI replace utility

Replace

Parameter	Description
`start-time`	Run the PI replace utility starting from this time. You can specify the time using an ISO date string with one of these formats: `2020-02-20`, `2020-02-20 10:05:00` or `2020-02-20 10:05:00.321`. You can also specify the time as a time span using one of these formats: `360:10:00:00` (UTC now - 360 days and 10 hours), `360:10:00:00.000`, `10:00:00` (UTC now - 10 hours), or `10:00:00.000`. The default value is the minimum CDF timestamp: `1971-01-01 00:00:00`
`end-time`	Run the PI replace utility ending at this time. You can specify the time using an ISO date string or a time span. See `start-time` for the allowed formats. The default value is UTC now.
`startup-delay`	Insert the delay in seconds before the PI replace utility starts. This delay doesn’t affect start and end times. The default value is 0.
`delete-data-points`	If you set this to `true`, the data points in the range are deleted from CDF before inserting data points from PI. The default value is `false`.
`throttling`	Enter the number of time series to replace concurrently. The default value is 10.
`step-hours`	Replace time series in iterations going back in time in steps of this many hours. The default will replace in steps of 24 hours. Set this to zero or less to disable replacing in iterations (replace the whole time range in a single iteration).
`save-progress-interval`	Enter an interval in seconds to save the PI replace progress to a state store. You must configure the selected storage in the State store section. The default value is 10.
`include-prefixes`, `include-tags`, `include-patterns`, `include-attribute-values`	Include time series based on these lists of tags (PI Point names), prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor. The default value is no value.
`exclude-prefixes`, `exclude-tags`, `exclude-patterns`, `exclude-attribute-values`	Exclude time series based on these lists of tags, prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor. The default value is no value.
`dry-run`	Run the PI replace utility in `dry-run` mode. The replace utility compares the data points in CDF with the data points in PI, but doesn’t upload any data points to CDF. The default value is `false`.
`timeout`	Enter a timeout in seconds for each replace iteration for a time series, for instance, to make sure a single iteration doesn’t take too long, forcing PI replace to stop. The default value is – 1 (disabled).
`status-codes`	Control which PI data points are ingested based on their status code.

Status codes

By default, the PI replace utility ingests only good data points and silently skips bad and uncertain data points. Include the status-codes parameter in the replace section to control which data points from the PI source system are written to CDF. You can configure the PI replace utility to ingest good and uncertain data points, or all data points regardless of status, similar to the PI extractor’s status-codes configuration.

Requires PI extractor 4.21.0 or later.

Parameter	Description
`status-codes-to-ingest`	Controls which PI data points are ingested based on their status code. Set to `All` to ingest all data points regardless of status, `Uncertain` to ingest good and uncertain data points (bad data points are skipped), or `GoodOnly` to ingest only good data points. The default value is `GoodOnly`.

Data quality

Include the data_quality section to compare the data points from PI with the data points in CDF to log and report any differences in the number of data points and the values of data points. If you’ve included the metrics section, the data quality metrics are sent to the configured Prometheus pushgateway. PI replace creates a CSV file with a summary by the end of the run.

Parameter	Description
`check-data-quality`	Set to `true` to enable data quality check. The replace utility compares data points from PI with data points in CDF and logs and reports differences. Data points are only replaced in CDF if differences are found. The default value is `false`.
`report-file-name`	Insert the name of the CSV file used to write the quality check reports. Leave this parameter empty if you don’t want to save the report to a file. The default value is `null`.
`ignore-positive-difference`	If CDF has more data points than PI, and you’ve set this parameter to `true`, the data points from CDF aren’t replaced with the data points from PI. The default value is `false`.

Events

You can configure the Cognite PI extractor to create CDF events for data loss and reconnection incidents. These incidents may cause data points to be missing in CDF. Include the events section to fetch these events from CDF at startup and replace the data points for the time range when these events occurred.

If you configure this section, the PI replace utility ignores the start-time and end-time parameters in the replace section.

Parameter	Description
`source`	Insert the name of the event source. This name must match the event source name configured in the PI extractor. The default value is `""`.
`external-id-prefix`	Fetch only events with this external ID prefix. This setting should match the external ID configured in the PI extractor. The default value is `""`.
`event-start-time`	Fetch events from CDF with a start time higher than or equal to this time. The allowed string formats are the same as the `start-time` parameter in the `replace` section. For example, 5:00:00:00 will fetch events starting 5 days ago. The default value is `""`.
`event-end-time`	Fetch events from CDF with an end time less than or equal to this time. For example, 00:20:00 will fetch events ending 20 minutes ago. The default value is `""`.
`expand-before`	Subtract this time span from the start time of the replace range (expand the range back in time). See the allowed time span formats in the `start-time` parameter in the replace section. The default value is `""`.
`expand-after`	Add this time span to the end time of the replace range (expand the range forward in time). The default value is `""`.

State store

Include the state-store and configure the save_progress_interval parameter in the replace section to store the PI replace progress. This is useful for long-running operations where a crash or restart may occur. PI replace will resume from where it stopped instead of starting again.

Parameter	Description
`database`	Enter the database you want to use. `LiteDb` creates a local file with the database. `Raw` will store data in CDF RAW. The default value is `none`.
`location`	Insert the database file path for `LiteDb` or database name for `Raw`. The default value is `""`.

​Minimal YAML configuration file

​Configure the PI replace utility

​Replace

​Status codes

​Data quality

​Events

​State store

Minimal YAML configuration file

Configure the PI replace utility

Replace

Status codes

Data quality

Events

State store