Configure the PI replace utility
To configure the PI replace utility, you must create a configuration file. The file must be in YAML format.
In addition to the sections listed below, you must include these PI extractor configuration sections:
Minimal YAML configuration file
The YAML settings below contain valid PI replace utility configurations, but not all configuration options are used. The values wrapped in ${} are replaced with environment variables with that name. For example, ${COGNITE_PROJECT} will be replaced with the value of the environment variable called COGNITE_PROJECT.
The configuration file also contains the global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 2 of the configuration schema.
version: 2
cognite:
project: ${COGNITE_PROJECT}
idp-authentication:
tenant: ${COGNITE_TENANT_ID}
client-id: ${COGNITE_CLIENT_ID}
secret: ${COGNITE_CLIENT_SECRET}
scopes:
- ${COGNITE_SCOPE}
time-series:
external-id-prefix: "pi:"
pi:
host: ${PI_HOST}
username: ${PI_USER}
password: ${PI_PASS}
metrics:
push-gateways:
- host: "http://localhost:9091"
job: "pi-replace-job"
state-store:
database: LiteDb
location: "PiReplaceStateStore.db"
logger:
console:
level: "information"
replace:
start-time: "2020-02-20 10:05:00"
end-time: "2020-03-20 10:05:00"
save-progress-interval: 10s
data-quality:
check-data-quality: true
report-file-name: "report"
ignore-positive-difference: true
Configure the PI replace utility
Replace
| Parameter | Description |
|---|---|
start-time | Run the PI replace utility starting from this time. You can specify the time using an ISO date string with one of these formats:2020-02-20, 2020-02-20 10:05:00 or 2020-02-20 10:05:00.321. You can also specify the time as a time span using one of these formats: 360:10:00:00 (UTC now - 360 days and 10 hours), 360:10:00:00.000, 10:00:00 (UTC now - 10 hours), or 10:00:00.000. The default value is the minimum CDF timestamp: 1971-01-01 00:00:00 |
end-time | Run the PI replace utility ending at this time. You can specify the time using an ISO date string or a time span. See start-time for the allowed formats. The default value is UTC now. |
startup-delay | Insert the delay in seconds before the PI replace utility starts. This delay doesn't affect start and end times. The default value is 0. |
delete-data-points | If you set this to true, the data points in the range are deleted from CDF before inserting data points from PI. The default value is false. |
throttling | Enter the number of time series to replace concurrently. The default value is 10. |
step-hours | Replace time series in iterations going back in time in steps of this many hours. The default will replace in steps of 24 hours. Set this to zero or less to disable replacing in iterations (replace the whole time range in a single iteration). |
save-progress-interval | Enter an interval in seconds to save the PI replace progress to a state store. You must configure the selected storage in the State store section. The default value is 10. |
include-prefixes, include-tags, include-patterns, include-attribute-values | Include time series based on these lists of tags (PI Point names), prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor. The default value is no value. |
exclude-prefixes, exclude-tags, exclude-patterns, exclude-attribute-values | Exclude time series based on these lists of tags, prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor. The default value is no value. |
dry-run | Run the PI replace utility in dry-run mode. The replace utility compares the data points in CDF with the data points in PI, but doesn't upload any data points to CDF. The default value is false. |
timeout | Enter a timeout in seconds for each replace iteration for a time series, for instance, to make sure a single iteration doesn't take too long, forcing PI replace to stop. The default value is – 1 (disabled). |
Data quality
Include the data_quality section to compare the data points from PI with the data points in CDF to log and report any differences in the number of data points and the values of data points. If you've included the metrics section, the data quality metrics are sent to the configured Prometheus pushgateway. PI replace creates a CSV file with a summary by the end of the run.
| Parameter | Description |
|---|---|
check-data-quality | Set to true to enable data quality check. The replace utility compares data points from PI with data points in CDF and logs and reports differences. Data points are only replaced in CDF if differences are found. The default value is false. |
report-file-name | Insert the name of the CSV file used to write the quality check reports. Leave this parameter empty if you don't want to save the report to a file. The default value is null. |
ignore-positive-difference | If CDF has more data points than PI, and you've set this parameter to true, the data points from CDF aren't replaced with the data points from PI. The default value is false. |
Events
You can configure the Cognite PI extractor to create CDF events for data loss and reconnection incidents. These incidents may cause data points to be missing in CDF. Include the events section to fetch these events from CDF at startup and replace the data points for the time range when these events occurred.
If you configure this section, the PI replace utility ignores the start-time and end-time parameters in the replace section.
| Parameter | Description |
|---|---|
source | Insert the name of the event source. This name must match the event source name configured in the PI extractor. The default value is ““. |
external-id-prefix | Fetch only events with this external ID prefix. This setting should match the external ID configured in the PI extractor. The default value is ““. |
event-start-time | Fetch events from CDF with a start time higher than or equal to this time. The allowed string formats are the same as the start-time parameter in the replace section. For example, 5:00:00:00 will fetch events starting 5 days ago. The default value is ““. |
event-end-time | Fetch events from CDF with an end time less than or equal to this time. For example, 00:20:00 will fetch events ending 20 minutes ago. The default value is ““. |
expand-before | Subtract this time span from the start time of the replace range (expand the range back in time). See the allowed time span formats in the start-time parameter in the replace section. The default value is ““. |
expand-after | Add this time span to the end time of the replace range (expand the range forward in time). The default value is ““. |
State store
Include the state-store and configure the save_progress_interval parameter in the replace section to store the PI replace progress. This is useful for long-running operations where a crash or restart may occur. PI replace will resume from where it stopped instead of starting again.
| Parameter | Description |
|---|---|
database | Enter the database you want to use. LiteDb creates a local file with the database. Raw will store data in CDF RAW. The default value is none. |
location | Insert the database file path for LiteDb or database name for Raw. The default value is ““. |