Creating a data model and mappings

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

In an asset-centric Cognite Data Fusion (CDF) project, assets, events, time series, and file metadata use fixed schemas. Data modeling lets you define a custom data model that reflects your organizational structure and data relationships. This guide walks data engineers through creating that data model and mappings from asset-centric resources.

Create a target data model that extends CDM

The data model must extend the CogniteCore model. See Create a data model.

Create resource view mappings

Map asset-centric properties to the target views. See Create resource view mappings.

Deploy the model and mappings

Deploy them to your CDF project so the migration plugin can use them. See Governance.

Create a data model

Consider using Neat to create your data model. Neat checks your data model for consistency and provides insights on how to improve it.

The migration plugin requires a target data model for your migration, but it does not prescribe how you create it. The data model must extend the CogniteCore model (CDM). That extension connects time series and files to their underlying CDF resources and lets you organize assets into hierarchies. A typical approach is to start from the CDM schema for assets, time series, files, and events and model unstructured data that lives in metadata fields. For example, the asset-centric Asset resource schema looks like this:

asset_schema.yaml

externalId: string
name: (required) string
parentExternalId: string
metadata: map<string, string>
description: string
dataSetId: integer
source: string
labels: list<label>
geoLocation: geoLocation

In the Kelmarsh example, structured values from metadata become explicit properties on a WindTurbine view:

windturbine.View.yaml

externalId: WindTurbine
space: sp_schema
version: v1
implements:
- externalId: CogniteAsset
  space: cdf_cdm
  type: view
  version: v1
properties:
  parent:
    container:
      externalId: CogniteAsset
      space: cdf_cdm
      type: container
    containerPropertyIdentifier: parent
    # We are overwriting the parent property to update the source
    source:
      type: view
      space: sp_schema
      externalId: KelmarshAsset
      version: v1
  model:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: model # Text type
  country:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: country # Text type
  latitude:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: latitude # Float64 type
  longitude:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: longitude # Float64 type
  manufacturer:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: manufacturer # Text type
  elevationM:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: elevationM # Float64 type
  hubHeightM:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: hubHeightM # Float64 type
  ratedPowerKW:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: ratedPowerKW # Int64 type
  alternativeTitle:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: alternativeTitle # Text type
  rotorDiameterM:
    container:
        externalId: WindTurbine
        space: sp_schema
        type: container
    containerPropertyIdentifier: rotorDiameterM # Int64 type
  commercialOperationsDate:
    container:
      externalId: WindTurbine
      space: sp_schema
      type: container
    containerPropertyIdentifier: commercialOperationsDate # Timestamp type

In the same way, create custom views for time series, files, and events as migration targets.

Create resource view mappings

Create a resource view mapping alongside the data model. The mapping defines how asset-centric properties map to properties in your target views. The Cognite Toolkit uses these mappings during migration to move data from asset-centric resources into the correct places in your data model, including data type conversion where needed. These mappings are YAML resources that the Cognite Toolkit can deploy to your CDF project:

windturbine.ResourceViewMapping.yaml

externalId: wind_turbine_mapping
resourceType: asset
viewId:
  space: sp_schema
  externalId: WindTurbine
  version: v1
propertyMapping:
  name: name
  description: description
  source: source # Note that this maps a string to a direct relation
  parentId: parent
  metadata.Model: model
  metadata.Country: country
  metadata.Latitude: latitude
  metadata.Longitude: longitude
  metadata.Manufacturer: manufacturer
  metadata.Elevation (m): elevationM
  metadata.Hub Height (m): hubHeightM
  metadata.Rated power (kW): ratedPowerKW
  metadata.Alternative Title: alternativeTitle
  metadata."Commercial Operations Date": commercialOperationsDate

After you create the data model and resource view mapping, deploy them to your CDF project. The migration plugin can then use the deployed model and mappings.

Governance

See the Data models and Resource view mapping sections of the Cognite Toolkit resource library for how to deploy and govern these resources.

​Create a data model

​Create resource view mappings

​Governance

​Further reading

Create a data model

Create resource view mappings

Governance

Further reading