Testing migration of production data in a development project

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

Migrating from an asset-centric to a data modeling-based CDF project is a significant change. Try the migration process in a development CDF project before running it in production. To do this, take a snapshot of the data and configuration in your production CDF project and restore it to a development CDF project. You can use the Cognite Toolkit data plugin for this. It is designed such that when you download data, you can use the same format to upload it again. The download will also include any configurations that you need, for example, if you download assets or events, the download will include data set and label resource configuration YAML files. These can then be deployed as part of the upload process (by using the --deploy-resources flag).

Replicate production data

To replicate production data to a development CDF project:

Get read access to production

You need read access to the data and configurations you will copy from the production CDF project.

Download production data

See Download data for supported data types. Download into a local folder, for example data. This can include assets, events, time series (with datapoints), files (with file content), Charts, Canvas, and 3D models.Store production credentials in a .env file and pass it with --env-path. For example, to download an asset hierarchy:

cdf --env-path prod.env data download hierarchy

After the download completes, you will have a folder structure like this:

data/
  resources/                  # Optional, only if deploy_resources is True
    raw/
      table1.Table.yaml
      table2.Table.yaml

  datafile1.<kind>.ndjson     # Data file of a specific kind
  datafile1.Manifest.yaml     # Manifest for datafile1

  datafile2-part-0001.<kind2>.ndjson   # Another data file
  datafile2-part-0002.<kind2>.ndjson   # Part 2 of data file
  datafile2.Manifest.yaml     # Manifest file for datafile2

Upload the data to the development project

Store development credentials in another .env file, for example dev.env. Upload the local data folder:

cdf --env-path dev.env data upload dir data/ --deploy-resources

With --deploy-resources, the Cognite Toolkit first deploys any configurations in the resources/ folder, then uploads the data.

Verify the copy

Confirm that the expected assets, events, files, time series, and application configurations exist in the development project before you run a migration command.

With production data now available in your development CDF project, you can safely test the migration process.

Deploy

Testing migration of production data in a development project

Replicate production data

Further reading

​Replicate production data

​Further reading

Replicate production data

Further reading