Migrating assets

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

This guide walks data engineers through migrating assets from an asset-centric Cognite Data Fusion (CDF) project to data modeling instances and recording lineage in the CogniteMigration data model.

Prerequisites

Before migrating assets, complete the following:

Required capabilities

Capability type	Action	Scope	Description
Assets	`assets:read`	The selected asset data sets	From data set where you read the source assets
Data model instances	`datamodelinstances:write`	The selected instance space	To space where you create the asset nodes
Data model instances	`datamodelinstances:write`	cognite_migration	To create the lineage nodes

Select assets by data set or by CSV file, then run cdf migrate assets twice: the first run creates assets and lineage; the second run establishes parent-child relationships.

You currently need to run this command twice. The first run creates the assets and their lineage. The second run establishes the parent-child relationships between the assets.

Choose how to select assets

Use data set selection if all assets go to the same view, mapping, and instance space. Use CSV file selection if you need different views, such as the root asset and wind turbines in the Kelmarsh example.

Run `cdf migrate assets`

Run the command in interactive mode, or pass the options as arguments. See cdf migrate assets --help.

cdf migrate assets

Run the command a second time so parent-child relationships are created after the assets exist.

Verify the migration

Confirm that lineage nodes exist in the InstanceSource view. See Verify the migration.

Data set selection

To migrate all assets in a data set, run the command in interactive mode:

cdf migrate assets

The command prompts you to select the source data set, resource view mapping, target instance space, and other options. You can also pass all options as command-line arguments. See cdf migrate assets --help for details. The disadvantage of data set selection is that all assets must use the same resource view mapping, target view, and instance space. In the Kelmarsh example, to migrate the root asset to a KelmarshAsset view and wind turbine assets to a WindTurbine view, use CSV file selection.

CSV file selection

To migrate assets using a CSV file, run:

cdf migrate assets --mapping-file <path to csv file>

Replace <path to csv file> with a path relative to your Cognite Toolkit project root. This CSV file migrates the Kelmarsh wind farm assets:

migration.Assets.csv

space,externalId,id,ingestionMapping,consumerViewSpace,consumerViewExternalId,consumerViewVersion
source_data,SEN 93420,2396603511069256,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,SEN 93425,3840956528416998,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,SEN 93424,5718533083684190,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,SEN 93422,6337175473438622,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,SEN 93423,7691815129534666,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,SEN 93421,8088410340821453,wind_turbine_asset_mapping,sp_schema,WindTurbine,v1
source_data,Kelmarsh,8997492773869554,cdf_asset_mapping,sp_schema,KelmarshAsset,v1

Required columns:

space: The target instance space where the asset will be created.
externalId: The external ID of the resulting asset.
id: The integer internal ID of the source asset to migrate.

Optional columns:

ingestionMapping: The resource view mapping to use. If omitted, the default mapping for assets is used (typically the CogniteAsset view in the CogniteCore model). The Cognite Toolkit reads mappings from the ResourceViewMapping view in the CogniteMigration data model. Deploy custom mappings with cdf build and cdf deploy using the Resource view mapping resource type.
consumerViewSpace, consumerViewExternalId, consumerViewVersion: The preferred consumer view for the asset. Canvas and Charts use this setting when they display the asset.

In the example above, wind turbine assets use the wind_turbine_asset_mapping mapping and WindTurbine consumer view. The root asset Kelmarsh uses the default cdf_asset_mapping mapping and KelmarshAsset consumer view.

Verify the migration

After a successful run, the Cognite Toolkit stores lineage for migrated assets in the InstanceSource view in the CogniteMigration data model. In the Kelmarsh example, lineage nodes for the wind turbine assets look like this:

InstanceSource lineage properties on WindTurbine nodes for Kelmarsh wind turbine assets in the migration plugin example.

The lineage nodes are the same instances as the created WindTurbine assets:

WindTurbine asset nodes in the Kelmarsh example after asset migration.

What `cdf migrate assets` does

When you run cdf migrate assets, the migration plugin performs the following steps on batches of assets:

Download assets

The Cognite Toolkit reads assets from the asset-centric /assets/byids or /assets/list endpoints, depending on the selection method.

Convert assets to instance nodes

The Cognite Toolkit looks up the mapping and converts each asset to an instance node. Mappings are defined by externalId in the ResourceViewMapping view in the CogniteMigration data model. The mapping for each asset comes from the ingestionMapping column in the CSV file or from the --ingestion-mapping CLI argument. If no mapping is specified, the default mapping is used.

Write instance nodes to CDF

The Cognite Toolkit writes the created instance nodes to CDF using the /models/instances endpoint.

Create lineage nodes

While writing instances, the Cognite Toolkit creates lineage nodes in the InstanceSource view in the CogniteMigration data model to record the lineage from source assets to the created instance nodes.

​Prerequisites

​Required capabilities