Source systems
This set of modules are data pipelines that extract data from a source system into CDF.
cdf_pi
This is an example of OSIsoft/AVEVA PI data source system ingestion into CDF.
Managed resources
- [RAW Table] — the staging area for the data from the source system.
- [Transformation] — moves the data from the RAW table to the CogniteTimeSeries.
- [Space] — the data space where the data is stored.
- [Group] — the group that controls the access of the workflow that executes the transformation.
- [DataSet] — controls the access to the transformation.
- [Workflow] — the workflow that runs the transformation that moves the data from the RAW table to the Cognite time series.
Variables
| Key | Description | Example Value |
|---|---|---|
| sourceName | The display name of the source. | Springfield AVEVA PI |
| sourceId | The identifier of the source. | springfield_aveva_pi |
| timeseriesTransformationExternalId | The identifier of the transformation. | pi_timeseries_springfield_aveva_pi |
| dataset | The dataset to control access to the transformation. | ingestion |
| rawSourceDatabase | The staging area database. | ingestion |
| schemaSpace | The schema space for the target CogniteTimeSeries. | cdf_cdm |
| instanceSpace | The data space to store the time series. | springfield_instances |
| groupSourceId | The source ID for the service principal that runs the workflow. | 12345-6789 |
| workflow | The name of the workflow to run the transformation. | aveva_pi_population |
| workflowClientId | The client ID for the service principal that runs the workflow. | ${AVEVA_PI_SPRINGFIELD_CLIENT_ID} |
| workflowClientSecret | The client secret for the service principal that runs the workflow. | ${AVEVA_PI_SPRINGFIELD_CLIENT_SECRET} |
The variable values ${AVEVA_PI_SPRINGFIELD_CLIENT_ID} and ${AVEVA_PI_SPRINGFIELD_CLIENT_SECRET} are
placeholders for environment variables. Use these for secrets and other sensitive information. In
the build process, the variable {{ workflowClientId }} is replaced with the ${AVEVA_PI_SPRINGFIELD_CLIENT_ID}
which in the deployment process will be replaced with the actual value of the environment variable.
cdf_sap_assets
This is an example of SAP source system asset/equipment data pipeline into CDF.
SAP source system events
This is an example of SAP source system event data pipeline into CDF.
Sharepoint files
This is an example of SharePoint source system file data pipeline into CDF.