> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Migrating data sets

> Step-by-step guide to migrate asset-centric data sets to instance spaces in a data modeling-based Cognite Data Fusion (CDF) project.

<Warning>
  The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.
</Warning>

This guide walks data engineers through migrating asset-centric **data sets** to instance spaces in Cognite Data Fusion (CDF) and recording lineage in the `CogniteMigration` data model.

## Prerequisites

Before migrating data sets, complete the following:

1. [Creating a data model and mappings](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/data_model_migration)
2. [Deploying the CogniteMigration data model](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/prepare)

## Required capabilities

| Capability type      | Action                     | Scope              | Description                   |
| -------------------- | -------------------------- | ------------------ | ----------------------------- |
| Data models          | `datamodels:write`         | All                | To create the instance spaces |
| Data model instances | `datamodelinstances:write` | cognite\_migration | To create the lineage node    |

## Migrating data sets

Data sets are migrated one-to-one to instance spaces. For each data set in the asset-centric model, the command creates a corresponding instance space in data modeling.

<Steps>
  <Step title="Run `cdf migrate data-sets`">
    Run the command in interactive mode to select which data sets to migrate, or pass the data sets and options as arguments. See `cdf migrate data-sets --help`.

    ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    cdf migrate data-sets
    ```
  </Step>

  <Step title="Verify the instance spaces and lineage">
    Confirm that an instance space exists for each selected data set, and that lineage nodes exist in the `CogniteMigration` data model. See [Verify the migration](#verify-the-migration).
  </Step>

  <Step title="Move the generated space files into a module">
    The command writes instance space configurations to `tmp/` (by default). Move them into the Cognite Toolkit `modules/` folder. See [Governing instance spaces](#governing-instance-spaces).
  </Step>
</Steps>

## Verify the migration

After a successful run, each selected data set has a corresponding instance space, and the `CogniteMigration` model contains lineage nodes for those data sets.

For example, in the Kelmarsh example project, migrating the data set produces this lineage node:

<Frame>
  <img class="illustration" src="https://apps-cdn.cogniteapp.com/@cognite/docs-portal-images/1.0.0/images/cdf/cdf_deploy/cdf_toolkit/migration-plugin/migration_dataset_source.png" alt="SpaceSource lineage node linking the source_data data set to the source_data instance space in the Kelmarsh example." width="100%" />
</Frame>

The lineage node shows that the instance space `source_data` was created from the data set with ID `1496359352069198` and
external ID `source_data`.

## Governing instance spaces

After you run `cdf migrate data-sets`, the Cognite Toolkit writes instance space configuration files to the `tmp/` folder (by default). Move these files into the Cognite Toolkit `modules/` folder and govern them going forward.

For the Kelmarsh example, move the files from `tmp/data_modeling/` to:

```shell wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
modules/
  kelmarsh/
    data_modeling/
      source_data.Space.yaml
```

The content of `source_data.Space.yaml` is:

```yaml title="source_data.Space.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
space: source_data
description: Source data for the data model
name: Source data
```

## What `cdf migrate data-sets` does

When you run `cdf migrate data-sets`, the migration plugin performs the following steps:

<Steps>
  <Step title="Download selected data sets">
    Download the selected data sets from your CDF project.
  </Step>

  <Step title="Convert each data set to an instance space">
    The `dataSetExternalId` becomes the `space` property of the instance space. The `space` value
    can only be 43 characters and must satisfy the regex `^[a-zA-Z][a-zA-Z0-9_-]{0,41}[a-zA-Z0-9]?$`, while
    data set external IDs can be up to 256 characters and can contain any UTF-8 characters. If you pass
    the `--auto-fix` flag, the migration plugin adjusts the `space` property to satisfy these requirements.
  </Step>

  <Step title="Create the new spaces">
    Create the instance spaces in your CDF project.
  </Step>

  <Step title="Create lineage nodes in the CogniteMigration data model">
    The lineage nodes are stored in the `SpaceSource` view in the `CogniteMigration` data model. They store the ID
    and external ID of the original data set, along with the ID of the created instance space.
    Later migration phases use this information to look up which instance space
    to use when creating data model instances.
  </Step>

  <Step title="Output the instance space configuration for each created instance space">
    The Cognite Toolkit writes the configuration to the `tmp/` folder by default. See [Governing instance spaces](#governing-instance-spaces).
  </Step>
</Steps>

## Further reading

* [Creating a data model and mappings](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/data_model_migration)
* [Migrating source systems](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/source_systems)
* [Kelmarsh example](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/example)
* [Migration overview](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/steps)