Migrating data sets

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

This guide walks data engineers through migrating asset-centric data sets to instance spaces in Cognite Data Fusion (CDF) and recording lineage in the CogniteMigration data model.

Prerequisites

Before migrating data sets, complete the following:

Required capabilities

Capability type	Action	Scope	Description
Data models	`datamodels:write`	All	To create the instance spaces
Data model instances	`datamodelinstances:write`	cognite_migration	To create the lineage node

Data sets are migrated one-to-one to instance spaces. For each data set in the asset-centric model, the command creates a corresponding instance space in data modeling.

Run `cdf migrate data-sets`

Run the command in interactive mode to select which data sets to migrate, or pass the data sets and options as arguments. See cdf migrate data-sets --help.

cdf migrate data-sets

Verify the instance spaces and lineage

Confirm that an instance space exists for each selected data set, and that lineage nodes exist in the CogniteMigration data model. See Verify the migration.

Move the generated space files into a module

The command writes instance space configurations to tmp/ (by default). Move them into the Cognite Toolkit modules/ folder. See Governing instance spaces.

Verify the migration

After a successful run, each selected data set has a corresponding instance space, and the CogniteMigration model contains lineage nodes for those data sets. For example, in the Kelmarsh example project, migrating the data set produces this lineage node:

SpaceSource lineage node linking the source_data data set to the source_data instance space in the Kelmarsh example.

The lineage node shows that the instance space source_data was created from the data set with ID 1496359352069198 and external ID source_data.

Governing instance spaces

After you run cdf migrate data-sets, the Cognite Toolkit writes instance space configuration files to the tmp/ folder (by default). Move these files into the Cognite Toolkit modules/ folder and govern them going forward. For the Kelmarsh example, move the files from tmp/data_modeling/ to:

modules/
  kelmarsh/
    data_modeling/
      source_data.Space.yaml

The content of source_data.Space.yaml is:

source_data.Space.yaml

space: source_data
description: Source data for the data model
name: Source data

What `cdf migrate data-sets` does

When you run cdf migrate data-sets, the migration plugin performs the following steps:

Download selected data sets

Download the selected data sets from your CDF project.

Convert each data set to an instance space

The dataSetExternalId becomes the space property of the instance space. The space value can only be 43 characters and must satisfy the regex ^[a-zA-Z][a-zA-Z0-9_-]{0,41}[a-zA-Z0-9]?$, while data set external IDs can be up to 256 characters and can contain any UTF-8 characters. If you pass the --auto-fix flag, the migration plugin adjusts the space property to satisfy these requirements.

Create the new spaces

Create the instance spaces in your CDF project.

Create lineage nodes in the CogniteMigration data model

The lineage nodes are stored in the SpaceSource view in the CogniteMigration data model. They store the ID and external ID of the original data set, along with the ID of the created instance space. Later migration phases use this information to look up which instance space to use when creating data model instances.

Output the instance space configuration for each created instance space

The Cognite Toolkit writes the configuration to the tmp/ folder by default. See Governing instance spaces.

Deploy

Migrating data sets

Prerequisites

Required capabilities