Migrating annotations

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

This guide walks data engineers through migrating file annotations from an asset-centric Cognite Data Fusion (CDF) project to data modeling instances and recording lineage in the CogniteMigration data model.

Prerequisites

Before migrating annotations, complete the following:

If you have not completed some prerequisites, see Manually populate the CogniteMigration data model.

Required capabilities

The following capabilities are required to run cdf migrate annotations:

Capability type	Action	Scope	Description
Files	`files:read`	The selected files data sets	From data set where you read the source files
Annotations	`annotations:read`	All	Read all annotations
Data model instances	`datamodelinstances:write`	The selected instance space	To space where you create the annotation nodes
Data model instances	`datamodelinstances:write`	cognite_migration	To create the lineage nodes

Only annotations of resource type file and annotation type diagrams.AssetLink or diagrams.FileLink can be migrated. See Migrating 360 images for 360° image annotation migration. Select annotations by data set or CSV file, then run cdf migrate annotations.

Choose how to select annotations

Use data set selection to migrate all annotations linked to files in a data set. Use CSV file selection to migrate a specific list of annotations.

Run `cdf migrate annotations`

Run the command in interactive mode, or pass the options as arguments. See cdf migrate annotations --help.

cdf migrate annotations

Verify the migration

Confirm that annotation nodes exist in the target instance space for the selected files. See Verify the migration.

Data set selection

To migrate all annotations linked to files in a data set, run the command in interactive mode:

cdf migrate annotations

The command prompts you to select the source data set (used to find the linked files and their annotations), and the target instance space where annotation nodes will be created. You can also pass all options as command-line arguments. See cdf migrate annotations --help for details.

CSV file selection

To migrate annotations using a CSV file, run:

cdf migrate annotations --mapping-file <path to csv file>

Replace <path to csv file> with a path relative to your Cognite Toolkit project root. This CSV file migrates an annotation from the Kelmarsh example:

migration.annotations.csv

space,externalId,id,ingestionView
source_data,newId,1245,cdf_asset_annotations_mapping

Required columns:

space: The target instance space where the annotation will be created.
externalId: The external ID of the resulting annotation.
id: The integer internal ID of the source annotation to migrate. This ID is returned in the response of the Filter annotations endpoint.

Optional columns:

ingestionView: The resource view mapping to use. This is the annotation-specific equivalent of the ingestionMapping column used in other migration CSV formats. If omitted, the default mapping for annotations is used (typically the CogniteDiagramAnnotation view in the CogniteCore model). Deploy custom mappings with cdf build and cdf deploy using the Resource view mapping resource type.

Verify the migration

After a successful run, confirm that annotation nodes exist in the target instance space. Query the instance space in CDF or use the DMS instances API to check that nodes with the expected externalId values are present. For data set selections, compare the annotation node count in the target instance space against the number of source annotations returned by the Filter annotations endpoint for the same data set.

What `cdf migrate annotations` does

When you run cdf migrate annotations, the migration plugin performs the following steps on batches of annotations:

Download annotations

The Cognite Toolkit reads annotations from the asset-centric annotations endpoint, filtering for resource type file and annotation types diagrams.AssetLink and diagrams.FileLink.

Convert annotations to instance nodes

The Cognite Toolkit applies the mapping and converts each annotation to an instance node. The mapping is read from the ingestionView column in the CSV file, or the default mapping is used if no mapping is specified.

Write instance nodes to CDF

The Cognite Toolkit writes the created instance nodes to CDF using the /models/instances endpoint.

Create lineage nodes

While writing instances, the Cognite Toolkit creates lineage nodes in the InstanceSource view in the CogniteMigration data model to record the lineage from source annotations to the created instance nodes.

​Prerequisites

​Required capabilities