Migrating events to records

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

This guide walks data engineers through migrating system-generated events from an asset-centric Cognite Data Fusion (CDF) project to records in a target stream using the Streams API.

Prerequisites

Before migrating events to records, complete the following:

Enable the migrate and records-migrate alpha flags in your cdf.toml file:

cdf.toml

# ...

[alpha_flags]
migrate = true
records-migrate = true

# ...

Provision a target stream with sufficient record and storage capacity. See Choose and size streams for planning guidance. cdf migrate events-to-records validates stream capacity before it runs and raises an error if the stream does not exist or has insufficient capacity.
Deploy one or more containers with usedFor: record in the target space.

Required capabilities

The following capabilities are required to run cdf migrate events-to-records:

Capability type	Action	Scope	Description
Events	`events:read`	Source events data sets	Read source events
Stream records	`streamrecords:read`	Target stream	Query existing records when using `--skip-existing`
Stream records	`streamrecords:write`	Target stream	Write migrated records to the stream
Streams	`streams:create`	All	Required only if the target stream needs to be created
Data models	`dataModels:read`	Space(s) where target containers are located	Resolve and validate target containers referenced in the config

Migrating events to records

Migrate human-created events (work orders, inspections, maintenance records) to data modeling nodes instead of records. See Migrating events.

Create a YAML config file, select events by data set or CSV file, then run cdf migrate events-to-records.

Create the YAML config file

Define the target stream and property mappings. See Config file.

Choose how to select events

Use data set selection with a defaultMapping, or CSV file selection to map specific events, such as status events in the Kelmarsh example.

Run `cdf migrate events-to-records`

Pass --config-file and either --data-set-id or --mapping-file, or run without selection flags for interactive mode. See cdf migrate events-to-records --help.

cdf migrate events-to-records --config-file <path to config yaml>

Verify the migration

Query the target stream and confirm that records exist for the selected events. See Verify the migration.

Config file

Unlike other migration commands, cdf migrate events-to-records requires a --config-file YAML that defines the target stream and property mappings. The config file names one or more mappings by externalId and maps source event fields to record container properties. Use defaultMapping when every event in a data set shares the same mapping, or when CSV rows omit ingestionMapping. Replace the placeholder values in the example below with your target stream, container, and field mappings.

events_to_records.config.yaml

streamExternalId: <target stream external id>
resourceType: event
defaultMapping: status_event_mapping   # optional; required when using --data-set-id
mappings:
  - externalId: status_event_mapping
    containerId:
      space: <container space>
      externalId: <container external id>
    propertyMapping:
      startTime: startTime
      endTime: endTime
      type: eventType
      description: description

streamExternalId: External ID of the target stream. The stream must already exist.
resourceType: Must be event.
defaultMapping: Mapping externalId to use when --data-set-id is set, or when a CSV row omits ingestionMapping.
mappings: Named mappings. propertyMapping maps source event field names to container property names.

Data set selection

To migrate all events in a data set, run:

cdf migrate events-to-records --config-file <path to config yaml> --data-set-id <data set id>

Replace <path to config yaml> with a path relative to your Cognite Toolkit project root. Replace <data set id> with the integer internal ID of the asset-centric source data set. The config file must include defaultMapping, because all events in the data set use the same mapping. To select the data set interactively, run:

cdf migrate events-to-records --config-file <path to config yaml>

The command prompts you to select the source data set. See cdf migrate events-to-records --help for details.

CSV file selection

To migrate events using a CSV mapping file, run:

cdf migrate events-to-records --config-file <path to config yaml> --mapping-file <path to csv file>

Replace <path to config yaml> and <path to csv file> with paths relative to your Cognite Toolkit project root. This CSV file migrates a Kelmarsh wind farm status event:

migration.events_to_records.csv

space,externalId,id,ingestionMapping
source_data,b67397de-9cab-4977-9c8d-08ebf6a2e207,2396603511069256,status_event_mapping

Required columns:

space: The space of the resulting record.
externalId: The external ID of the resulting record.
id: The integer internal ID of the source event to migrate.

Optional columns:

ingestionMapping: Mapping externalId from the config file for this row. If omitted, the config file defaultMapping is used. Unlike cdf migrate events, there are no consumerViewSpace, consumerViewExternalId, or consumerViewVersion columns.

Verify the migration

Query the target stream and confirm that records exist for the migrated events. In CDF, open the stream in Data management, or query record count through the Streams API. Use --skip-existing to query the target stream before uploading and skip records that already exist. This helps resume a partially completed migration without creating duplicate records. See cdf migrate events-to-records --help for the full list of options, including --dry-run, --log-dir, --yes, and --verbose.

What `cdf migrate events-to-records` does

When you run cdf migrate events-to-records, the migration plugin performs the following steps on batches of events:

Download events

The Cognite Toolkit reads events from the asset-centric /events/byids or /events/list endpoints, depending on the selection method.

Map event fields to record properties

The Cognite Toolkit applies the mapping from the config file mappings section. propertyMapping maps source event field names to target container property names. Each event uses the mapping from the CSV ingestionMapping column, the --data-set-id defaultMapping, or the config file defaultMapping.

Write records to the target stream

The Cognite Toolkit writes migrated records to the target stream through the Streams API.

Skip existing records (optional)

When --skip-existing is set, the Cognite Toolkit queries the target stream first and skips records that already exist.

​Prerequisites

​Required capabilities

​Migrating events to records

​Config file

​Data set selection

​CSV file selection

​Verify the migration

​What cdf migrate events-to-records does

​Further reading

Prerequisites

Required capabilities

Migrating events to records

Config file

Data set selection

CSV file selection

Verify the migration

What `cdf migrate events-to-records` does

Further reading