> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Migrating events to records

> Step-by-step guide to migrate system-generated events from an asset-centric Cognite Data Fusion (CDF) project to records using the Streams API.

<Warning>
  The migration plugin is an experimental feature. Updates to the Cognite Toolkit are likely to introduce changes to the described commands and processes.
</Warning>

This guide walks data engineers through migrating system-generated **events** from an asset-centric Cognite Data Fusion (CDF) project to **records** in a target stream using the [Streams API](/cdf/dm/records/concepts/records_and_streams).

## Prerequisites

Before migrating events to records, complete the following:

1. Enable the `migrate` and `records-migrate` alpha flags in your `cdf.toml` file:

```toml cdf.toml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
# ...

[alpha_flags]
migrate = true
records-migrate = true

# ...
```

2. Provision a [target stream](/cdf/dm/records/concepts/records_and_streams#stream-templates) with sufficient record and storage capacity. See [Choose and size streams](/cdf/dm/records/guides/choose_and_size_streams) for planning guidance. `cdf migrate events-to-records` validates stream capacity before it runs and raises an error if the stream does not exist or has insufficient capacity.

3. Deploy one or more [containers](/cdf/dm/dm_concepts/dm_containers_views_datamodels#which-types-of-instances-can-you-use-a-container-for) with `usedFor: record` in the target space.

## Required capabilities

The following capabilities are required to run `cdf migrate events-to-records`:

| Capability type | Action                | Scope                                        | Description                                                     |
| --------------- | --------------------- | -------------------------------------------- | --------------------------------------------------------------- |
| Events          | `events:read`         | Source events data sets                      | Read source events                                              |
| Stream records  | `streamrecords:read`  | Target stream                                | Query existing records when using `--skip-existing`             |
| Stream records  | `streamrecords:write` | Target stream                                | Write migrated records to the stream                            |
| Streams         | `streams:create`      | All                                          | Required only if the target stream needs to be created          |
| Data models     | `dataModels:read`     | Space(s) where target containers are located | Resolve and validate target containers referenced in the config |

## Migrating events to records

<Note>
  Migrate human-created events (work orders, inspections, maintenance records) to data modeling nodes instead of records. See [Migrating events](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/events).
</Note>

Create a YAML config file, select events by data set or CSV file, then run `cdf migrate events-to-records`.

<Steps>
  <Step title="Create the YAML config file">
    Define the target stream and property mappings. See [Config file](#config-file).
  </Step>

  <Step title="Choose how to select events">
    Use [data set selection](#data-set-selection) with a `defaultMapping`, or [CSV file selection](#csv-file-selection) to map specific events, such as status events in the [Kelmarsh example](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/example).
  </Step>

  <Step title="Run `cdf migrate events-to-records`">
    Pass `--config-file` and either `--data-set-id` or `--mapping-file`, or run without selection flags for interactive mode. See `cdf migrate events-to-records --help`.

    ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    cdf migrate events-to-records --config-file <path to config yaml>
    ```
  </Step>

  <Step title="Verify the migration">
    Query the target stream and confirm that records exist for the selected events. See [Verify the migration](#verify-the-migration).
  </Step>
</Steps>

## Config file

Unlike other migration commands, `cdf migrate events-to-records` requires a `--config-file` YAML that defines the target stream and property mappings.

The config file names one or more mappings by `externalId` and maps source event fields to record container properties. Use `defaultMapping` when every event in a data set shares the same mapping, or when CSV rows omit `ingestionMapping`.

Replace the placeholder values in the example below with your target stream, container, and field mappings.

```yaml title="events_to_records.config.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
streamExternalId: <target stream external id>
resourceType: event
defaultMapping: status_event_mapping   # optional; required when using --data-set-id
mappings:
  - externalId: status_event_mapping
    containerId:
      space: <container space>
      externalId: <container external id>
    propertyMapping:
      startTime: startTime
      endTime: endTime
      type: eventType
      description: description
```

* `streamExternalId`: External ID of the target stream. The stream must already exist.
* `resourceType`: Must be `event`.
* `defaultMapping`: Mapping `externalId` to use when `--data-set-id` is set, or when a CSV row omits `ingestionMapping`.
* `mappings`: Named mappings. `propertyMapping` maps source event field names to container property names.

## Data set selection

To migrate all events in a data set, run:

```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf migrate events-to-records --config-file <path to config yaml> --data-set-id <data set id>
```

Replace `<path to config yaml>` with a path relative to your Cognite Toolkit project root. Replace `<data set id>` with the integer **internal ID** of the asset-centric source data set. The config file must include `defaultMapping`, because all events in the data set use the same mapping.

To select the data set interactively, run:

```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf migrate events-to-records --config-file <path to config yaml>
```

The command prompts you to select the source data set. See `cdf migrate events-to-records --help` for details.

## CSV file selection

To migrate events using a CSV mapping file, run:

```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf migrate events-to-records --config-file <path to config yaml> --mapping-file <path to csv file>
```

Replace `<path to config yaml>` and `<path to csv file>` with paths relative to your Cognite Toolkit project root.

This CSV file migrates a Kelmarsh wind farm status event:

```csv title="migration.events_to_records.csv" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
space,externalId,id,ingestionMapping
source_data,b67397de-9cab-4977-9c8d-08ebf6a2e207,2396603511069256,status_event_mapping
```

Required columns:

* `space`: The space of the resulting record.
* `externalId`: The external ID of the resulting record.
* `id`: The integer **internal ID** of the source event to migrate.

Optional columns:

* `ingestionMapping`: Mapping `externalId` from the config file for this row. If omitted, the config file `defaultMapping` is used. Unlike `cdf migrate events`, there are no `consumerViewSpace`, `consumerViewExternalId`, or `consumerViewVersion` columns.

## Verify the migration

Query the target stream and confirm that records exist for the migrated events. In CDF, open the stream in **Data management**, or query record count through the [Streams API](/api-reference/concepts/20230101/streams).

Use `--skip-existing` to query the target stream before uploading and skip records that already exist. This helps resume a partially completed migration without creating duplicate records.

See `cdf migrate events-to-records --help` for the full list of options, including `--dry-run`, `--log-dir`, `--yes`, and `--verbose`.

## What `cdf migrate events-to-records` does

When you run `cdf migrate events-to-records`, the migration plugin performs the following steps on batches of events:

<Steps>
  <Step title="Download events">
    The Cognite Toolkit reads events from the asset-centric `/events/byids` or `/events/list` endpoints, depending on the selection method.
  </Step>

  <Step title="Map event fields to record properties">
    The Cognite Toolkit applies the mapping from the config file `mappings` section. `propertyMapping` maps source event field names to target container property names.
    Each event uses the mapping from the CSV `ingestionMapping` column, the `--data-set-id` `defaultMapping`, or the config file `defaultMapping`.
  </Step>

  <Step title="Write records to the target stream">
    The Cognite Toolkit writes migrated records to the target stream through the Streams API.
  </Step>

  <Step title="Skip existing records (optional)">
    When `--skip-existing` is set, the Cognite Toolkit queries the target stream first and skips records that already exist.
  </Step>
</Steps>

## Further reading

* [Migrating events](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/events)
* [Migrating files and time series](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/files_timeseries)
* [Kelmarsh example](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/example)
* [Migration overview](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/steps)