CDF Entity Matching reference

The CDF Entity Matching deployment pack (dp:contextualization:cdf_entity_matching) automates the contextualization of entities — typically time series — to a configurable target view (Asset, Tags, or Equipment) in Cognite Data Fusion. It deploys as a Cognite Toolkit module and uses a CDF Workflow to run two CDF Functions in sequence: a metadata update step that enriches aliases and applies NORSOK-based discipline classification, followed by an entity matching step that uses that enriched metadata to establish direct relationships in the data model. The matching step ships production-tested code, combining manual expert mappings, rule-based matching with regex, and AI-powered entity matching in a single pipeline, with RAW-backed state for incremental runs.

CDF Entity Matching is a cherry-pickable module (dp:contextualization:cdf_entity_matching) inside the Contextualization deployment pack (dp:contextualization). You can add it on its own or alongside the other contextualization modules.

Who it’s for

Target personas:

Data engineers — deploy, configure, and tune the matching pipeline.
Solution architects — design the contextualization approach across source systems.

Use this pack when:

You need to match time series (or another entity view) to assets automatically, without building a custom matching pipeline from scratch.
Your project combines data from multiple sources (for example, SAP assets and OSIsoft/AVEVA PI time series) and requires relationship discovery across them.
You need a combination of manual expert mappings, rule-based logic, and AI-powered matching in a single orchestrated workflow.
You need incremental processing — only new or updated entities should be matched on each run.
You want a scalable foundation proven for 10,000+ time series per batch in production environments.

When not to use this pack:

If your assets and time series are already contextualized and you only need to maintain relationships — this pack is optimized for initial and incremental matching, not purely for maintenance.
If your project requires document or P&ID annotation, use the P&ID Annotation deployment pack instead.

What’s included

Component	Description	Optional?
Metadata update function (`fn_dm_context_metadata_update`)	Updates aliases on time series and assets and applies NORSOK-based discipline classification to improve match quality, using caching and batch processing.	Yes
Entity matching function (`fn_dm_context_timeseries_entity_matching`)	Matches entity instances to the target view using manual expert mappings, rule-based regex, and AI-powered (ML) matching, with retry logic and state management.	No
Workflow (`EntityMatching`)	Runs the metadata update function, then the entity matching function (`dependsOn`), each with 3 retries, a 9000-second timeout, and `abortWorkflow` on failure.	No
Scheduled trigger (`EntityMatching_trigger`)	Cron trigger that runs the workflow daily at 04:00 UTC using the configured workflow credentials.	Yes
Extraction pipelines	`ep_ctx_entity_matching_metadata_update` and `ep_ctx_timeseries_<location>_<source>_entity_matching` hold the runtime parameters and view configuration each function reads.	No
RAW database and tables	`db_asset_entity_matching` with `contextualization_manual_input`, `contextualization_rule_input`, `contextualization_good`, `contextualization_bad`, and `contextualization_state_store`.	No
`upload_data/` directory	Seed manual mappings (CSV) and rule definitions (NDJSON) uploaded to RAW after deployment with `cdf data upload`.	No
Data modeling resources	Function space (`sp_entity_matching_fn`) plus `match_type` and `sourceSystem` nodes.	No
Access group and dataset	A processing access group bound to a configured IdP group source ID, and a time series dataset.	No

Prerequisites

Verify all of the following before you start:

Cognite Toolkit version 0.8 or later is installed. See Setting up.
A cdf.toml file exists in your project root. If it is missing, run cdf modules init and select Create toml file (required).
The data plugin is enabled in cdf.toml — required for the cdf data upload step after deployment:
[plugins] data = true
Authentication is configured and verified — run cdf auth init and cdf auth verify using a local .env file. See Authentication and authorization.
A data model with the target view (for example, Asset) and entity view (for example, TimeSeries) is deployed, and instances are already ingested.
You have a service principal (app registration) in your identity provider (IdP) for the scheduled workflow to authenticate as. Have its client ID and client secret ready — these populate IDP_CLIENT_ID / IDP_CLIENT_SECRET in your .env (mapped to workflowClientId / workflowClientSecret).
You have an IdP group for the processing access group, and you know its source ID — the group’s object ID in the IdP (populates GROUP_SOURCE_ID, mapped to entity_matching_processing_group_source_id). The workflow’s service principal must be a member of this group.

Install and deploy

You can add the Entity Matching module to an existing Toolkit project or start a new project from scratch.

Add to an existing project
Start a new project

Add the module from your project directory:

cdf modules add -d cdf_entity_matching

In an empty directory, initialize a project:
cdf modules init .
In the interactive menu, choose Contextualization, use Space to select Contextualization Entity Matching, then press Enter to confirm.

The Toolkit creates the module under modules/contextualization/cdf_entity_matching/ and updates your config.<env>.yaml files. Before building, open config.dev.yaml and set at minimum project, the schema and instance spaces, the target and entity view external IDs and search properties, and the credentials (workflowClientId / workflowClientSecret and entity_matching_processing_group_source_id) — see Configuration reference. Then build and deploy:

Build deployment artifacts

cdf build

The Toolkit substitutes template variables and writes artifacts to the build/ directory. If you need to change the project name or any variables marked <change_me>, edit config.<env>.yaml first — see Configure, build, and deploy modules.

Dry-run the deployment

cdf deploy --dry-run

Inspect the output and confirm that configurations look correct before deploying.

Deploy to CDF

cdf deploy

The Toolkit deploys only configurations that have changed since the last run.

Set up CI/CD (optional)

For governed production deployments, automate build, dry-run, and deploy in a pipeline. See Set up CI/CD pipelines.

After deploying, populate the RAW tables with the seed manual mappings and rule definitions:

cdf data upload dir ./modules/contextualization/cdf_entity_matching/upload_data/ -v

The data plugin must be enabled in cdf.toml (see Prerequisites) before running cdf data upload.

After the upload, your project should include:

modules/
└── contextualization/
    └── cdf_entity_matching/

Verify the deployment

After deploying, confirm success with these checks:

The cdf deploy command finishes without errors and reports successful deployment of Functions, Workflows, Extraction Pipelines, RAW, the data set, the access group, and the data modeling resources.
The cdf data upload command finishes without errors, and contextualization_manual_input and contextualization_rule_input are visible in Integrate > Staging (RAW) under db_asset_entity_matching.
In CDF, navigate to Integrate > Data Workflows and verify that the EntityMatching workflow is listed.
Trigger the workflow with Run (or wait for the daily EntityMatching_trigger). Confirm that the metadata update task completes before the entity matching task, and that both finish without errors.
Navigate to Build Solutions > Functions, open fn_dm_context_metadata_update and fn_dm_context_timeseries_entity_matching, select View logs, and review the metrics (processed/updated counts, cache hit rate, match scores).
Navigate to Integrate > Staging (RAW) and verify that contextualization_good and contextualization_bad are populated.
Use CDF Search or Data Modeling to confirm that direct relationships exist between matched entities and the target view (when dmUpdate is enabled).

The deployment is successful when the workflow completes without errors, the good and bad match tables are populated, and direct relationships exist between matched entities.

Configuration reference

Module variables are set per environment in config.<env>.yaml. Update these before running cdf build. Each variable takes its default unless you override it. Where the default is —, a placeholder (for example YourOrg, ORG), or a ${...} value read from .env, you must provide your own value before deploying.

When this module is installed as part of the Foundation Demo bundle, several of these values (for example, the target and entity view external IDs and search properties) are overridden to match the Quickstart data model. Follow the Foundation Demo instructions in that case.

Variable	Description	Default
`project`	CDF project external ID to deploy to.	—
`schemaSpace`	Schema space containing the target and entity views.	`sp_enterprise_process_industry`
`viewVersion`	Version of the target and entity views.	`v1`
`assetInstanceSpace`	Instance space for the target (asset) instances.	`springfield_instances`
`timeseriesInstanceSpace`	Instance space for the entity (time series) instances.	`springfield_instances`
`targetViewExternalId`	External ID of the target view to match to (for example, `Asset`).	`YourOrgAsset`
`entityViewExternalId`	External ID of the entity view to match from (for example, `TimeSeries`).	`YourOrgTimeSeries`
`targetViewSearchProperty`	Target property used to compute match candidates.	`name`
`entityViewSearchProperty`	Entity property used to compute match candidates.	`aliases`
`targetViewFilterValues`	`tags` values scoping the target instances.	`["root:WMT"]`
`entityViewFilterValues`	`tags` values scoping the entity instances.	`["site:VAL"]`
`AssetViewExternalId` / `TimeSeriesViewExternalId`	Views the metadata update step enriches.	`YourOrgAsset` / `YourOrgTimeSeries`
`functionSpace`	Space where the function code nodes are stored.	`sp_entity_matching_fn`
`dbName`	RAW database for state, inputs, and results.	`db_asset_entity_matching`
`workflow`	Workflow external ID.	`EntityMatching`
`workflowClientId` / `workflowClientSecret`	IdP credentials used by the scheduled trigger.	`${IDP_CLIENT_ID}` / `${IDP_CLIENT_SECRET}`
`entity_matching_processing_group_source_id`	IdP group source ID for the processing access group.	`${GROUP_SOURCE_ID}`
`organization`, `location_name`, `source_name`	Naming inputs used in resource external IDs (for example, the matching pipeline ID).	`ORG`, `Springfield`, `springfield`
`reservedWordPrefix`	Prefix applied to reserved-word view external IDs.	`Enterprise_`

Runtime parameters (extraction pipelines)

Matching behavior is controlled by the extraction pipeline configs, not config.<env>.yaml. Edit extraction_pipelines/ctx_timeseries_entity_matching.config.yaml to tune the matching run.

Parameter	Description	Default
`autoApprovalThreshold`	Confidence at or above this value is auto-approved and written to `contextualization_good`; below it goes to `contextualization_bad`.	`0.85`
`runAll`	Match all entities (`true`) or only those updated since the last run (`false`).	`true` (shipped config)
`dmUpdate`	Write approved matches as Data Model relationships (`true`) or to RAW only (`false`).	`true` (shipped config)
`removeOldLinks`	Remove existing target links before applying new matches.	`false`
`debug`	Process a single entity for debugging.	`false`

If you change autoApprovalThreshold, clean up the relationships created by previous runs before re-running. Only matches created by this process are removed; manually created relationships are preserved.

Architecture

The pack deploys two CDF Functions into a dedicated function space, orchestrated by the EntityMatching workflow and a daily trigger. Each function reads its runtime parameters from an extraction pipeline config. RAW holds the manual and rule inputs, the good and bad match results, and the incremental state.

Data flow

The workflow runs the metadata update function first (to improve alias quality), then the matching function. The matching function applies three methods in priority order — manual mappings, then rule-based regex, then ML matching — and splits results by autoApprovalThreshold.

Matching pipeline

The matching function (fn_dm_context_timeseries_entity_matching) executes in this order:

Read configuration and the RAW input tables (manual mappings, rule mappings).
Apply manual mappings from entity to target — these overwrite any existing mapping.
Read the entities not yet matched (or all entities when runAll is true).
Read the target view instances (for example, assets).
Run rule-based matching using the configured regex patterns.
Run ML entity matching in CDF for the remaining candidates.
Update entity→target relationships in the data model when dmUpdate is true.
Write results to contextualization_good (score ≥ autoApprovalThreshold) and contextualization_bad (below threshold).

Incremental processing and state management

State is maintained in contextualization_state_store. With runAll: false, only entities updated since the last run are processed; previously matched entities are skipped. Set runAll: true (the shipped default) or clear the state table to reprocess everything.

Resilience

The metadata update and entity matching tasks each retry up to 3 times with a 9000-second timeout, and the workflow aborts if a task ultimately fails. The matching function processes entities in batches with bounded exponential-backoff retries on Data Modeling writes, and external IDs prevent duplicate relationships across runs.

Troubleshooting

Symptom	Likely cause	Resolution
Workflow fails with an access error	The workflow credentials or processing group are misconfigured.	Verify `workflowClientId` / `workflowClientSecret` and `entity_matching_processing_group_source_id` in `config.<env>.yaml` point at the correct IdP application and group source ID. See Access management.
`cdf data upload` fails with “data plugin not enabled”	`[plugins] data = true` is missing from `cdf.toml`.	Add `data = true` under `[plugins]` in `cdf.toml` and retry the upload command.
No matches created after the workflow run	Input tables are missing, or search properties do not align with the data model.	Verify that `cdf data upload` populated `contextualization_manual_input` and `contextualization_rule_input`, and that `targetViewSearchProperty` / `entityViewSearchProperty` match real property names.
No matches created after the workflow run	Filter criteria in the configuration do not match any data.	Check the extraction pipeline configuration and edit the filters to match your data, or remove the `filterProperty` and `filterValues` config lines.
No matches created after the workflow run	The test data’s views, schema space, instance space, or version do not match the configuration.	Check the extraction pipeline configuration and make sure it matches your data in CDF.
Matches written to RAW but no Data Model relationships	`dmUpdate` is set to `false`.	Set `dmUpdate: true` in `extraction_pipelines/ctx_timeseries_entity_matching.config.yaml` and re-run.
Duplicate or stale relationships after threshold change	`autoApprovalThreshold` changed without cleaning up prior matches.	Remove relationships created by previous runs (or set `removeOldLinks: true`) before re-running. Manually created relationships are preserved.
Workflow processes all entities on every run	`runAll` is `true`, or the state table was cleared.	Set `runAll: false` for incremental runs. Confirm `contextualization_state_store` exists and is being updated.
Low match rate despite correct configuration	Metadata enrichment did not complete, or aliases are missing.	Check `fn_dm_context_metadata_update` logs and confirm aliases and descriptions exist before the matching task runs.
`cdf deploy` fails with “Insufficient permissions”	Toolkit service account is missing required CDF capabilities.	Grant capabilities to deploy Functions, Workflows, Extraction Pipelines, RAW, Data Sets, spaces, and groups.

For issues not covered here, contact Cognite support.

Support

Field	Details
Team	Cognite Value Delivery Team
Documentation	Cognite product documentation
Questions and feedback	Deployment Packs group on Cognite Hub
Version	1.0.0
Last updated	2026

​Who it’s for

​What’s included

​Prerequisites

​Install and deploy

​Verify the deployment

​Configuration reference

​Runtime parameters (extraction pipelines)

​Architecture

​Data flow

​Matching pipeline

​Incremental processing and state management

​Resilience

​Troubleshooting

​Support

Who it’s for

What’s included

Prerequisites

Install and deploy

Verify the deployment

Configuration reference

Runtime parameters (extraction pipelines)

Architecture

Data flow

Matching pipeline

Incremental processing and state management

Resilience

Troubleshooting

Support