Migrating events

The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.

This guide walks data engineers through migrating events from an asset-centric Cognite Data Fusion (CDF) project to data modeling instances and recording lineage in the CogniteMigration data model.

Prerequisites

Before migrating events, complete the following:

Required capabilities

The following capabilities are required to run cdf migrate events:

Capability type	Action	Scope	Description
Events	`events:read`	The selected events data sets	From data set where you read the source events
Data model instances	`datamodelinstances:write`	The selected instance space	To space where you create the event nodes
Data model instances	`datamodelinstances:write`	cognite_migration	To create the lineage nodes

Migrate human-created events (work orders, inspections, maintenance records) to data modeling nodes with this procedure. System-generated events (alarms, sensor events, logs) should be migrated to records instead. See Migrating events to records.

Select events by data set or by CSV file, then run cdf migrate events.

Choose how to select events

Use data set selection if all events go to the same view, mapping, and instance space. Use CSV file selection to map events to different views, such as status events in the Kelmarsh example.

Run `cdf migrate events`

Run the command in interactive mode, or pass the options as arguments. See cdf migrate events --help.

cdf migrate events

Verify the migration

Confirm that lineage nodes exist in the InstanceSource view. See Verify the migration.

Data set selection

To migrate all events in a data set, run the command in interactive mode:

cdf migrate events

The command prompts you to select the source data set, resource view mapping, target instance space, and other options. You can also pass all options as command-line arguments. See cdf migrate events --help for details. The disadvantage of data set selection is that all events must use the same resource view mapping, target view, and instance space. Use CSV file selection when events need different mappings or consumer views.

CSV file selection

To migrate events using a CSV file, run:

cdf migrate events --mapping-file <path to csv file>

Replace <path to csv file> with a path relative to your Cognite Toolkit project root. This CSV file migrates a Kelmarsh wind farm status event:

migration.events.csv

space,externalId,id,ingestionMapping,consumerViewSpace,consumerViewExternalId,consumerViewVersion
source_data,b67397de-9cab-4977-9c8d-08ebf6a2e207,2396603511069256,status_event_mapping,sp_schema,StatusEvent,v1

Required columns:

space: The target instance space where the event will be created.
externalId: The external ID of the resulting event.
id: The integer internal ID of the source event to migrate.

Optional columns:

ingestionMapping: The resource view mapping to use. If omitted, the default mapping for events is used (typically the CogniteActivity view in the CogniteCore model). The Cognite Toolkit reads mappings from the ResourceViewMapping view in the CogniteMigration data model. Deploy custom mappings with cdf build and cdf deploy using the Resource view mapping resource type.
consumerViewSpace, consumerViewExternalId, consumerViewVersion: The preferred consumer view for the event. Canvas and Charts use this setting when they display the event.

In the example above, the status event uses the status_event_mapping mapping and StatusEvent consumer view.

Verify the migration

After a successful run, the Cognite Toolkit stores lineage for migrated events in the InstanceSource view in the CogniteMigration data model. In the Kelmarsh example, lineage nodes for the status events look like this:

InstanceSource lineage properties on a StatusEvent node for a Kelmarsh status event in the migration plugin example.

The lineage nodes are the same instances as the created StatusEvent nodes:

StatusEvent nodes in the Kelmarsh example after event migration.

What `cdf migrate events` does

When you run cdf migrate events, the migration plugin performs the following steps on batches of events:

Download events

The Cognite Toolkit reads events from the asset-centric /events/byids or /events/list endpoints, depending on the selection method.

Convert events to instance nodes

The Cognite Toolkit looks up the mapping and converts each event to an instance node. Mappings are defined by externalId in the ResourceViewMapping view in the CogniteMigration data model. The mapping for each event comes from the ingestionMapping column in the CSV file or from the --ingestion-mapping CLI argument. If no mapping is specified, the default mapping is used.

Write instance nodes to CDF

The Cognite Toolkit writes the created instance nodes to CDF using the /models/instances endpoint.

Create lineage nodes

While writing instances, the Cognite Toolkit creates lineage nodes in the InstanceSource view in the CogniteMigration data model to record the lineage from source events to the created instance nodes.

​Prerequisites

​Required capabilities