source field on assets, events, and files in Cognite Data Fusion (CDF).
Prerequisites
Before migrating source systems, complete the following:Required capabilities
The following capabilities are required to runcdf migrate source-systems:
Source systems in data modeling
A source system is a data modeling concept that did not exist in the asset-centric model. It is part of the CogniteCore data model. A source system represents the source of the data, such as a specific ERP system, SCADA system, or any other data source. In the asset-centric model, assets, events, and files could be tagged with a text field calledsource
to indicate the source of the data. For example, an asset could have a source field with the value “SCADA System A”.
asset_schema.yaml
Migrating source systems
The Cognite Toolkit reads uniquesource values from assets, events, and files in the selected data sets and creates a CogniteSourceSystem node for each value.
1
Run `cdf migrate source-systems`
Run the command in interactive mode to select which data sets to read
source values from, or pass the data sets and options as arguments. See cdf migrate source-systems --help.2
Verify the source system nodes
Confirm that each unique
source value has a CogniteSourceSystem node and a matching lineage node. See Verify the migration.3
Move the generated node files into a module
The command writes source system nodes to
tmp/ (by default). Move them into the Cognite Toolkit modules/ folder. See Governing source system nodes.Verify the migration
For example, in the Kelmarsh example project, migrating thezenodo source produces this lineage node:
zenodo was migrated to a CogniteSourceSystem in space source_data with external ID zenodo.
Deploy all
CogniteSourceSystem nodes before creating any nodes that reference them. The CogniteSourceable container has a required constraint that the referenced source system must already exist.Governing source system nodes
After you runcdf migrate source-systems, the Cognite Toolkit writes source system node files to the tmp/ folder (by default). Move these files into the Cognite Toolkit modules/ folder and govern them going forward.
For the Kelmarsh example, move the files from tmp/data_modeling/ to:
zenodo.Node.yaml is:
zenodo.Node.yaml
What cdf migrate source-systems does
When you run cdf migrate source-systems, the Cognite Toolkit performs the following steps:
1
Read source property values
The Cognite Toolkit reads all unique values stored in the
source property on assets, events, and files in the selected data sets.
It uses the respective /aggregate endpoint to read unique values. That endpoint does not preserve
casing, so if you have the same source value with different casing, you end up with one source system with the
value lowercased. For example, source values SCADA System A and scada system a produce one source system with the value scada system a.2
Create source system nodes
For each unique value, the Cognite Toolkit creates a
CogniteSourceSystem node in the target instance space with the
externalId and name set to the source value. The space of the new nodes is set by the --instance-space
argument when you run the command.3
Create lineage nodes
For each created source system, the Cognite Toolkit creates a lineage node in the
CogniteMigration data model that
represents the migration of the source system. The lineage node connects to the created source system
and stores the original source value in a property.4
Output source system node files
The Cognite Toolkit writes node configuration files to the
tmp/ folder by default. See Governing source system nodes.