> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.
# CDF Entity Matching reference
> Automate contextualization of time series to assets in CDF with a workflow that combines metadata enrichment, manual mappings, rule-based regex, and AI-powered entity matching.
The **CDF Entity Matching deployment pack** (`dp:contextualization:cdf_entity_matching`) automates the contextualization of entities — typically time series — to a configurable target view (`Asset`, `Tags`, or `Equipment`) in Cognite Data Fusion. It deploys as a Cognite Toolkit module and uses a CDF Workflow to run two CDF Functions in sequence: a **metadata update** step that enriches aliases and applies NORSOK-based discipline classification, followed by an **entity matching** step that uses that enriched metadata to establish direct relationships in the data model.
The matching step ships production-tested code, combining **manual expert mappings, rule-based matching with regex, and AI-powered entity matching** in a single pipeline, with RAW-backed state for incremental runs.
CDF Entity Matching is a cherry-pickable module (`dp:contextualization:cdf_entity_matching`) inside the **Contextualization** deployment pack (`dp:contextualization`). You can add it on its own or alongside the other contextualization modules.
## Who it's for
**Target personas:**
* **Data engineers** — deploy, configure, and tune the matching pipeline.
* **Solution architects** — design the contextualization approach across source systems.
**Use this pack when:**
* You need to match time series (or another entity view) to assets automatically, without building a custom matching pipeline from scratch.
* Your project combines data from multiple sources (for example, SAP assets and OSIsoft/AVEVA PI time series) and requires relationship discovery across them.
* You need a combination of manual expert mappings, rule-based logic, and AI-powered matching in a single orchestrated workflow.
* You need incremental processing — only new or updated entities should be matched on each run.
* You want a scalable foundation proven for 10,000+ time series per batch in production environments.
**When not to use this pack:**
* If your assets and time series are already contextualized and you only need to maintain relationships — this pack is optimized for initial and incremental matching, not purely for maintenance.
* If your project requires document or P\&ID annotation, use the [P\&ID Annotation](/cdf/deploy/cdf_toolkit/references/packages/pid_annotation) deployment pack instead.
## What's included
| Component | Description | Optional? |
| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| Metadata update function (`fn_dm_context_metadata_update`) | Updates aliases on time series and assets and applies NORSOK-based discipline classification to improve match quality, using caching and batch processing. | Yes |
| Entity matching function (`fn_dm_context_timeseries_entity_matching`) | Matches entity instances to the target view using manual expert mappings, rule-based regex, and AI-powered (ML) matching, with retry logic and state management. | No |
| Workflow (`EntityMatching`) | Runs the metadata update function, then the entity matching function (`dependsOn`), each with 3 retries, a 9000-second timeout, and `abortWorkflow` on failure. | No |
| Scheduled trigger (`EntityMatching_trigger`) | Cron trigger that runs the workflow daily at 04:00 UTC using the configured workflow credentials. | Yes |
| Extraction pipelines | `ep_ctx_entity_matching_metadata_update` and `ep_ctx_timeseries___entity_matching` hold the runtime parameters and view configuration each function reads. | No |
| RAW database and tables | `db_asset_entity_matching` with `contextualization_manual_input`, `contextualization_rule_input`, `contextualization_good`, `contextualization_bad`, and `contextualization_state_store`. | No |
| `upload_data/` directory | Seed manual mappings (CSV) and rule definitions (NDJSON) uploaded to RAW after deployment with `cdf data upload`. | No |
| Data modeling resources | Function space (`sp_entity_matching_fn`) plus `match_type` and `sourceSystem` nodes. | No |
| Access group and dataset | A processing access group bound to a configured IdP group source ID, and a time series dataset. | No |
## Prerequisites
Verify all of the following before you start:
* Cognite Toolkit version **0.8 or later** is installed. See [Setting up](/cdf/deploy/cdf_toolkit/guides/setup).
* A `cdf.toml` file exists in your project root. If it is missing, run `cdf modules init` and select **Create toml file (required)**.
* The data plugin is enabled in `cdf.toml` — required for the `cdf data upload` step after deployment:
```toml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
[plugins]
data = true
```
* Authentication is configured and verified — run `cdf auth init` and `cdf auth verify` using a local `.env` file. See [Authentication and authorization](/cdf/deploy/cdf_toolkit/guides/auth).
* A data model with the target view (for example, `Asset`) and entity view (for example, `TimeSeries`) is deployed, and instances are already ingested.
* You have a service principal (app registration) in your identity provider (IdP) for the scheduled workflow to authenticate as. Have its client ID and client secret ready — these populate `IDP_CLIENT_ID` / `IDP_CLIENT_SECRET` in your `.env` (mapped to `workflowClientId` / `workflowClientSecret`).
* You have an IdP group for the processing access group, and you know its **source ID** — the group's object ID in the IdP (populates `GROUP_SOURCE_ID`, mapped to `entity_matching_processing_group_source_id`). The workflow's service principal must be a member of this group.
## Install and deploy
You can add the Entity Matching module to an existing Toolkit project or start a new project from scratch.
Add the module from your project directory:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules add -d cdf_entity_matching
```
1. In an empty directory, initialize a project:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules init .
```
2. In the interactive menu, choose **Contextualization**, use **Space** to select **Contextualization Entity Matching**, then press **Enter** to confirm.
The Toolkit creates the module under `modules/contextualization/cdf_entity_matching/` and updates your `config..yaml` files. Before building, open `config.dev.yaml` and set at minimum `project`, the schema and instance spaces, the target and entity view external IDs and search properties, and the credentials (`workflowClientId` / `workflowClientSecret` and `entity_matching_processing_group_source_id`) — see [Configuration reference](#configuration-reference). Then build and deploy:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf build
```
The Toolkit substitutes template variables and writes artifacts to the `build/` directory. If you need to change the project name or any variables marked ``, edit `config..yaml` first — see [Configure, build, and deploy modules](/cdf/deploy/cdf_toolkit/guides/usage).
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy --dry-run
```
Inspect the output and confirm that configurations look correct before deploying.
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy
```
The Toolkit deploys only configurations that have changed since the last run.
For governed production deployments, automate build, dry-run, and deploy in a pipeline. See [Set up CI/CD pipelines](/cdf/deploy/cdf_toolkit/guides/cicd/index).
After deploying, populate the RAW tables with the seed manual mappings and rule definitions:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf data upload dir ./modules/contextualization/cdf_entity_matching/upload_data/ -v
```
The data plugin must be enabled in `cdf.toml` (see [Prerequisites](#prerequisites)) before running `cdf data upload`.
After the upload, your project should include:
```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
modules/
└── contextualization/
└── cdf_entity_matching/
```
## Verify the deployment
After deploying, confirm success with these checks:
1. The `cdf deploy` command finishes without errors and reports successful deployment of Functions, Workflows, Extraction Pipelines, RAW, the data set, the access group, and the data modeling resources.
2. The `cdf data upload` command finishes without errors, and `contextualization_manual_input` and `contextualization_rule_input` are visible in **Integrate > Staging (RAW)** under `db_asset_entity_matching`.
3. In CDF, navigate to **Integrate > Data Workflows** and verify that the `EntityMatching` workflow is listed.
4. Trigger the workflow with **Run** (or wait for the daily `EntityMatching_trigger`). Confirm that the metadata update task completes before the entity matching task, and that both finish without errors.
5. Navigate to **Build Solutions > Functions**, open `fn_dm_context_metadata_update` and `fn_dm_context_timeseries_entity_matching`, select **View logs**, and review the metrics (processed/updated counts, cache hit rate, match scores).
6. Navigate to **Integrate > Staging (RAW)** and verify that `contextualization_good` and `contextualization_bad` are populated.
7. Use CDF Search or Data Modeling to confirm that direct relationships exist between matched entities and the target view (when `dmUpdate` is enabled).
The deployment is successful when the workflow completes without errors, the good and bad match tables are populated, and direct relationships exist between matched entities.
## Configuration reference
Module variables are set per environment in `config..yaml`. Update these before running `cdf build`.
Each variable takes its default unless you override it. Where the default is `—`, a placeholder (for example `YourOrg`, `ORG`), or a `${...}` value read from `.env`, you must provide your own value before deploying.
When this module is installed as part of the [Foundation Demo](/cdf/deploy/cdf_toolkit/references/packages/quickstart) bundle, several of these values (for example, the target and entity view external IDs and search properties) are overridden to match the Quickstart data model. Follow the Foundation Demo instructions in that case.
| Variable | Description | Default |
| -------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------- |
| `project` | CDF project external ID to deploy to. | — |
| `schemaSpace` | Schema space containing the target and entity views. | `sp_enterprise_process_industry` |
| `viewVersion` | Version of the target and entity views. | `v1` |
| `assetInstanceSpace` | Instance space for the target (asset) instances. | `springfield_instances` |
| `timeseriesInstanceSpace` | Instance space for the entity (time series) instances. | `springfield_instances` |
| `targetViewExternalId` | External ID of the target view to match to (for example, `Asset`). | `YourOrgAsset` |
| `entityViewExternalId` | External ID of the entity view to match from (for example, `TimeSeries`). | `YourOrgTimeSeries` |
| `targetViewSearchProperty` | Target property used to compute match candidates. | `name` |
| `entityViewSearchProperty` | Entity property used to compute match candidates. | `aliases` |
| `targetViewFilterValues` | `tags` values scoping the target instances. | `["root:WMT"]` |
| `entityViewFilterValues` | `tags` values scoping the entity instances. | `["site:VAL"]` |
| `AssetViewExternalId` / `TimeSeriesViewExternalId` | Views the metadata update step enriches. | `YourOrgAsset` / `YourOrgTimeSeries` |
| `functionSpace` | Space where the function code nodes are stored. | `sp_entity_matching_fn` |
| `dbName` | RAW database for state, inputs, and results. | `db_asset_entity_matching` |
| `workflow` | Workflow external ID. | `EntityMatching` |
| `workflowClientId` / `workflowClientSecret` | IdP credentials used by the scheduled trigger. | `${IDP_CLIENT_ID}` / `${IDP_CLIENT_SECRET}` |
| `entity_matching_processing_group_source_id` | IdP group source ID for the processing access group. | `${GROUP_SOURCE_ID}` |
| `organization`, `location_name`, `source_name` | Naming inputs used in resource external IDs (for example, the matching pipeline ID). | `ORG`, `Springfield`, `springfield` |
| `reservedWordPrefix` | Prefix applied to reserved-word view external IDs. | `Enterprise_` |
### Runtime parameters (extraction pipelines)
Matching behavior is controlled by the extraction pipeline configs, not `config..yaml`. Edit `extraction_pipelines/ctx_timeseries_entity_matching.config.yaml` to tune the matching run.
| Parameter | Description | Default |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `autoApprovalThreshold` | Confidence at or above this value is auto-approved and written to `contextualization_good`; below it goes to `contextualization_bad`. | `0.85` |
| `runAll` | Match all entities (`true`) or only those updated since the last run (`false`). | `true` (shipped config) |
| `dmUpdate` | Write approved matches as Data Model relationships (`true`) or to RAW only (`false`). | `true` (shipped config) |
| `removeOldLinks` | Remove existing target links before applying new matches. | `false` |
| `debug` | Process a single entity for debugging. | `false` |
If you change `autoApprovalThreshold`, clean up the relationships created by previous runs before re-running. Only matches created by this process are removed; manually created relationships are preserved.
## Architecture
The pack deploys two CDF Functions into a dedicated function space, orchestrated by the `EntityMatching` workflow and a daily trigger. Each function reads its runtime parameters from an extraction pipeline config. RAW holds the manual and rule inputs, the good and bad match results, and the incremental state.
### Data flow
The workflow runs the metadata update function first (to improve alias quality), then the matching function. The matching function applies three methods in priority order — manual mappings, then rule-based regex, then ML matching — and splits results by `autoApprovalThreshold`.
```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
flowchart TD
Trigger["Scheduled trigger EntityMatching_trigger · daily 04:00 UTC"] --> WF["EntityMatching workflow"]
WF --> Meta["fn_dm_context_metadata_update aliases + NORSOK discipline · caching"]
Meta --> Match["fn_dm_context_timeseries_entity_matching"]
TargetView["Target view (e.g. Asset)"] --> Match
EntityView["Entity view (e.g. TimeSeries)"] --> Match
Manual[("RAW · contextualization_manual_input")] --> Match
Rules[("RAW · contextualization_rule_input")] --> Match
Match --> M1["1 · Manual expert mappings"]
M1 --> M2["2 · Rule-based regex"]
M2 --> M3["3 · ML entity matching"]
M3 --> Score{"score ≥ autoApprovalThreshold (0.85)?"}
Score -->|yes| Good[("RAW · contextualization_good")]
Score -->|no| Bad[("RAW · contextualization_bad")]
Good -->|dmUpdate = true| DM["Direct relationships entity → target (Data Modeling)"]
State[("RAW · contextualization_state_store")] -.->|skip unchanged, incremental| Match
Match -.->|update state| State
```
### Matching pipeline
The matching function (`fn_dm_context_timeseries_entity_matching`) executes in this order:
1. Read configuration and the RAW input tables (manual mappings, rule mappings).
2. Apply **manual mappings** from entity to target — these overwrite any existing mapping.
3. Read the entities not yet matched (or all entities when `runAll` is `true`).
4. Read the target view instances (for example, assets).
5. Run **rule-based** matching using the configured regex patterns.
6. Run **ML entity matching** in CDF for the remaining candidates.
7. Update entity→target relationships in the data model when `dmUpdate` is `true`.
8. Write results to `contextualization_good` (score ≥ `autoApprovalThreshold`) and `contextualization_bad` (below threshold).
### Incremental processing and state management
State is maintained in `contextualization_state_store`. With `runAll: false`, only entities updated since the last run are processed; previously matched entities are skipped. Set `runAll: true` (the shipped default) or clear the state table to reprocess everything.
### Resilience
The metadata update and entity matching tasks each retry up to 3 times with a 9000-second timeout, and the workflow aborts if a task ultimately fails. The matching function processes entities in batches with bounded exponential-backoff retries on Data Modeling writes, and external IDs prevent duplicate relationships across runs.
## Troubleshooting
| Symptom | Likely cause | Resolution |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Workflow fails with an access error | The workflow credentials or processing group are misconfigured. | Verify `workflowClientId` / `workflowClientSecret` and `entity_matching_processing_group_source_id` in `config..yaml` point at the correct IdP application and group source ID. See [Access management](/cdf/access). |
| `cdf data upload` fails with "data plugin not enabled" | `[plugins] data = true` is missing from `cdf.toml`. | Add `data = true` under `[plugins]` in `cdf.toml` and retry the upload command. |
| No matches created after the workflow run | Input tables are missing, or search properties do not align with the data model. | Verify that `cdf data upload` populated `contextualization_manual_input` and `contextualization_rule_input`, and that `targetViewSearchProperty` / `entityViewSearchProperty` match real property names. |
| No matches created after the workflow run | Filter criteria in the configuration do not match any data. | Check the extraction pipeline configuration and edit the filters to match your data, or remove the `filterProperty` and `filterValues` config lines. |
| No matches created after the workflow run | The test data's views, schema space, instance space, or version do not match the configuration. | Check the extraction pipeline configuration and make sure it matches your data in CDF. |
| Matches written to RAW but no Data Model relationships | `dmUpdate` is set to `false`. | Set `dmUpdate: true` in `extraction_pipelines/ctx_timeseries_entity_matching.config.yaml` and re-run. |
| Duplicate or stale relationships after threshold change | `autoApprovalThreshold` changed without cleaning up prior matches. | Remove relationships created by previous runs (or set `removeOldLinks: true`) before re-running. Manually created relationships are preserved. |
| Workflow processes all entities on every run | `runAll` is `true`, or the state table was cleared. | Set `runAll: false` for incremental runs. Confirm `contextualization_state_store` exists and is being updated. |
| Low match rate despite correct configuration | Metadata enrichment did not complete, or aliases are missing. | Check `fn_dm_context_metadata_update` logs and confirm aliases and descriptions exist before the matching task runs. |
| `cdf deploy` fails with "Insufficient permissions" | Toolkit service account is missing required CDF capabilities. | Grant capabilities to deploy Functions, Workflows, Extraction Pipelines, RAW, Data Sets, spaces, and groups. |
For issues not covered here, contact Cognite support.
## Support
| Field | Details |
| ---------------------- | --------------------------------------------------------------------------- |
| Team | Cognite Value Delivery Team |
| Documentation | [Cognite product documentation](/) |
| Questions and feedback | [Deployment Packs group on Cognite Hub](https://hub.cognite.com/groups/578) |
| Version | 1.0.0 |
| Last updated | 2026 |