CDF Entity Matching reference

The CDF Entity Matching deployment pack (dp:contextualization:cdf_entity_matching) automates the contextualization of entities, typically time series, to a configurable target view (for example, CogniteAsset, Tags, or Equipment) in Cognite Data Fusion (CDF). The pack deploys as a module for the Cognite Toolkit. A data workflow runs two functions in Cognite Functions:

An aliases update function that writes normalized aliases on time series, assets, and files.
An entity matching function that runs in a submit stage and a collect stage to establish direct relationships in the data model.

The matching function ships production-tested code that combines manual expert mappings, rule-based matching with regex, and AI-powered entity matching in a single pipeline, with state stored in CDF RAW for incremental runs. Splitting matching into submit and collect stages means a long-running prediction job doesn’t time out the function.

CDF Entity Matching is a cherry-pickable module (dp:contextualization:cdf_entity_matching) inside the Contextualization deployment pack (dp:contextualization). You can add it on its own or alongside the other contextualization modules.

Who it’s for

Target personas:

Data engineers — deploy, configure, and tune the matching pipeline.
Solution architects — design the contextualization approach across source systems.

Use this pack when:

You need to match time series (or another entity view) to assets automatically, without building a custom matching pipeline from scratch.
Your project combines data from multiple sources (for example, SAP assets and OSIsoft/AVEVA PI time series), possibly in separate instance spaces, and requires relationship discovery across them.
Source system tags are formatted differently from asset names, and you need normalized aliases so matching compares like-for-like tags.
You need a combination of manual expert mappings, rule-based logic, and AI-powered matching in a single orchestrated workflow.
You need incremental processing, where only new or updated entities are matched on each run.
You want a scalable foundation proven for 10,000+ time series per batch in production environments.

When not to use this pack:

If your assets and time series are already contextualized and you only need to maintain relationships. This pack is optimized for initial and incremental matching, not purely for maintenance.
If your project requires document or piping and instrumentation diagram (P&ID) annotation, use the P&ID Annotation deployment pack instead.

What’s included

Component	Description	Optional?
Aliases update function (`fn_dm_context_aliases_update`)	Writes normalized `aliases` on time series, assets, and (optionally) files, derived from each instance’s `name` with configurable regex patterns.	Yes
Entity matching function (`fn_dm_context_entity_matching`)	One function with two stages selected by the `stage` input. Submit applies manual and rule-based mappings and starts the entity matching predict job. Collect polls queued jobs, merges results, and writes matches to RAW and the data model.	No
Data workflow (`wf_ctx_entity_matching`)	Runs aliases update, then submit, then collect. Each task has a 540-second timeout and `abortWorkflow` on failure. Aliases update and collect retry up to 3 times; submit is not retried.	No
Scheduled trigger (`wf_ctx_entity_matching_trigger`)	Cron trigger that runs the workflow using the configured workflow credentials. The default schedule only fires on 29 February, so runs are started manually until you set `workflowSchedule`.	Yes
Extraction pipelines	`ep_ctx_aliases_update` and `ep_ctx_entity_matching` hold the runtime parameters and view configuration each function reads.	No
RAW database and tables	`db_asset_entity_matching` with `contextualization_manual_input`, `contextualization_rule_input`, `contextualization_good`, `contextualization_bad`, and `contextualization_state_store`.	No
`upload_data/` directory	Seed manual mappings (CSV) and rule definitions in newline-delimited JSON (NDJSON) uploaded to RAW after deployment with `cdf data upload`.	No
Data modeling resources	Function space (`sp_entity_matching_fn`) and a `CogniteSourceSystem` node that identifies the entity matcher.	No
Access group and data set	The `gp_entity_matching_processing` access group bound to a configured identity provider (IdP) group source ID, and the `ds_entity_matching` data set for the workflow, extraction pipelines, and the function’s cache and staging files.	No
`testing/` directory	Optional test-only `site`/`unit` scope model and transformations for trying scoped matching. Not deployed by default.	Yes

Prerequisites

Verify all of the following before you start:

The Cognite Toolkit version 0.8 or later is installed. See Setting up.
A cdf.toml file exists in your project root. If it’s missing, run cdf modules init and select Create toml file (required).
The data plugin is enabled in cdf.toml. You need it for the cdf data upload step after deployment:
[plugins] data = true
Authentication is configured and verified. Run cdf auth init and cdf auth verify using a local .env file. See Authentication and authorization.
A data model with views that implement CogniteAsset and CogniteTimeSeries (or the core data model views themselves) is deployed, and instances are already ingested.
You have a service principal (app registration) in your IdP for the scheduled workflow to authenticate as. Have its client ID and client secret ready. These populate IDP_CLIENT_ID and IDP_CLIENT_SECRET in your .env file, mapped to workflowClientId and workflowClientSecret.
You have an IdP group for the processing access group, and you know its source ID, the group’s object ID in the IdP. This populates GROUP_SOURCE_ID, mapped to entity_matching_processing_group_source_id. The workflow’s service principal must be a member of this group.

Install and deploy

You can add the Entity Matching module to an existing Cognite Toolkit project or start a new project from scratch.

Add to an existing project
Start a new project

Add the module from your project directory:

cdf modules add -d cdf_entity_matching

In an empty directory, initialize a project:
cdf modules init .
In the interactive menu, choose Contextualization, use Space to select Contextualization Entity Matching, then press Enter to confirm.

The Cognite Toolkit creates the module under modules/contextualization/cdf_entity_matching/ and updates your config.<env>.yaml files. Before building, open config.dev.yaml and set at minimum:

project.
The schema and instance spaces.
The target and entity view external IDs and search properties.
The alias patterns.
The credentials: workflowClientId, workflowClientSecret, and entity_matching_processing_group_source_id.

See Configuration reference for every variable. Then build and deploy:

Build deployment artifacts

cdf build

The Toolkit substitutes template variables and writes artifacts to the build/ directory. If you need to change the project name or any variables marked <change_me>, edit config.<env>.yaml first — see Configure, build, and deploy modules.

Dry-run the deployment

cdf deploy --dry-run

Inspect the output and confirm that configurations look correct before deploying.

Deploy to CDF

cdf deploy

The Toolkit deploys only configurations that have changed since the last run.

Set up CI/CD (optional)

For governed production deployments, automate build, dry-run, and deploy in a pipeline. See Set up CI/CD pipelines.

After deploying, populate the RAW tables with the seed manual mappings and rule definitions:

cdf data upload dir ./modules/contextualization/cdf_entity_matching/upload_data/ -v

The data plugin must be enabled in cdf.toml (see Prerequisites) before running cdf data upload.

After the upload, your project should include:

modules/
└── contextualization/
    └── cdf_entity_matching/
        ├── auth/
        ├── data_modeling/
        ├── data_sets/
        ├── extraction_pipelines/
        ├── functions/
        │   ├── fn_dm_context_aliases_update/
        │   └── fn_dm_context_entity_matching/
        ├── raw/
        ├── testing/
        ├── upload_data/
        ├── workflows/
        ├── default.config.yaml
        └── module.toml

Verify the deployment

After deploying, confirm success with these checks:

The cdf deploy command finishes without errors and reports successful deployment of Functions, Workflows, Extraction Pipelines, RAW, the data set, the access group, and the data modeling resources.
The cdf data upload command finishes without errors, and contextualization_manual_input and contextualization_rule_input are visible in Integrate > Staging (RAW) under db_asset_entity_matching.
In CDF, navigate to Integrate > Data Workflows and verify that the wf_ctx_entity_matching workflow is listed.
Trigger the workflow with Run. The default schedule rarely fires, so start the first run manually. Confirm that the aliases update, submit, and collect tasks run in that order and finish without errors.
Navigate to Build Solutions > Functions, open fn_dm_context_aliases_update and fn_dm_context_entity_matching, select View logs, and review the processed and updated counts and match scores.
Use CDF Search or Data Modeling to confirm that time series and assets have the expected aliases values.
Navigate to Integrate > Staging (RAW) and verify that contextualization_good and contextualization_bad are populated.
Confirm that direct relationships exist between matched entities and the target view (when dmUpdate is enabled).

The deployment is successful when the workflow completes without errors, the good and bad match tables are populated, and direct relationships exist between matched entities.

If a predict job is still running when the collect task’s polling budget runs out, it stays queued and the next workflow run collects it. On a first run with many entities, run the workflow again if the good and bad tables are empty.

Configuration reference

Set module variables per environment in config.<env>.yaml before running cdf build. Required variables must be set for your project. Optional variables can keep their defaults. Where the default is — or a ${...} value read from .env, you must provide your own value before deploying. Runtime parameters live in the extraction pipeline configs under extraction_pipelines/, and the functions read them on every run. The settings are grouped by what they control:

General settings (this section): project, spaces, workflow, credentials, and function input.
Aliases update settings: the views, patterns, and parameters for fn_dm_context_aliases_update.
Entity matching settings: the views, filters, scopes, inputs, and parameters for fn_dm_context_entity_matching.

When this module is installed as part of the Foundation Demo bundle, several of these values (for example, the target and entity view external IDs and search properties) are overridden to match the Quickstart data model. Follow the Foundation Demo instructions in that case.

General variables

These variables apply to the whole module:

Variable	Type	Required	Description	Default
`project`	string	Required	CDF project external ID to deploy to.	—
`function_version`	string	Optional	Version stored in the function metadata.	`v1.0.0`
`functionOwner`	string	Optional	Team or person shown as the functions’ owner in CDF.	`cdf_entity_matching`
`schemaSpace`	string	Optional	Schema space containing the target, entity, asset, and time series views.	`cdf_cdm`
`viewVersion`	string	Optional	Version of those views.	`v1`
`functionSpace`	string	Optional	Space for the function code nodes and the source system node. Created by this module.	`sp_entity_matching_fn`
`dbName`	string	Optional	RAW database for state, inputs, and results.	`db_asset_entity_matching`
`contextualizationId` / `contextualizationName`	string	Optional	External ID and name of the `CogniteSourceSystem` node for the matcher.	`entity_matcher` / `Entity Matcher`
`workflow`	string	Optional	Data workflow external ID. The trigger is named `<workflow>_trigger`.	`wf_ctx_entity_matching`
`workflowSchedule`	string	Optional	Cron expression for the scheduled trigger. The default only fires on 29 February. Set, for example, `'0 2 * * *'` for a nightly run.	`'0 0 29 2 *'`
`workflowClientId` / `workflowClientSecret`	string	Required	IdP credentials used by the scheduled trigger.	`${IDP_CLIENT_ID}` / `${IDP_CLIENT_SECRET}`
`entity_matching_processing_group_source_id`	string	Required	IdP group source ID for the processing access group.	`${GROUP_SOURCE_ID}`

Multiple instance spaces

Both functions read and write instances in these spaces. Each variable takes a single space or a list of spaces:

Variable	Type	Required	Description	Default
`assetInstanceSpace`	string or list of strings	Optional	Space or spaces for the target (asset) instances.	`inst_location`
`timeseriesInstanceSpace`	string or list of strings	Optional	Space or spaces for the entity (time series) instances.	`inst_location`
`fileInstanceSpace`	string or list of strings	Optional	Space or spaces for the file instances that aliases update enriches.	`inst_location`

assetInstanceSpace: inst_location
fileInstanceSpace: inst_documents
timeseriesInstanceSpace:
  - inst_timeseries_pi
  - inst_timeseries_sap

Instances are read from every listed space, and both functions write each instance back to the space it was read from. Matching isn’t restricted by space, so a time series in one space can match an asset in another. This lets per-source time series spaces work against a shared asset space. An external ID is unique within a space, not across spaces. Instances are identified by space and external ID together, so copies with the same external ID in different spaces are matched, linked, and updated independently. Two things are still keyed on external ID alone:

Manual mappings: the manual mapping RAW table has only TsExternalId and AssetExternalId columns, so a manual mapping applies to every copy of that external ID.
Asset links: when an asset external ID exists in several of the spaces in assetInstanceSpace, the space a link points to is arbitrary.

Entity matching logs a warning with examples whenever it reads duplicated external IDs, separately for entities and assets. Use unique external IDs per space, or configure a single space, to avoid both cases. When more than one time series space is configured, the space becomes part of the row key in the good and bad RAW tables so copies don’t overwrite each other. With a single space, the key is the external ID. If an instance’s space can’t be determined, the link is written to the first space in the list and a warning names the instance. That happens, for example, when a manual mapping names a target outside the configured filter.

Function input

The workflow passes these values in each function call’s input data. Both functions validate them before calling CDF. These are task inputs, not config.<env>.yaml variables.

Field	Type	Required	Description	Default
`ExtractionPipelineExtId`	string	Required	Extraction pipeline the function reads its config from: `ep_ctx_aliases_update` or `ep_ctx_entity_matching`.	—
`stage`	string	Required	Used by `fn_dm_context_entity_matching`. Either `submit` or `collect`. The aliases update function doesn’t use this field.	—
`logLevel`	string	Optional	Logging level for a workflow task: `DEBUG`, `INFO`, `WARNING`, or `ERROR`. Use `DEBUG` when troubleshooting.	`INFO`

Aliases update settings

These settings control how fn_dm_context_aliases_update derives normalized aliases from instance names. Entity matching uses those aliases when you set the search properties to aliases.

Use this function as the only aliases function in the project. The File Annotation module matches drawings on the same aliases and doesn’t write its own. Keep fileView enabled if you annotate files. When you change assetAliasPattern or fileAliasPattern, update entityNormalizationPatterns and fileNormalizationPatterns in File Annotation to match.

Aliases update variables

Variable	Type	Required	Description	Default
`assetViewExternalId` / `timeseriesViewExternalId`	string	Optional	Asset and time series views that aliases update enriches.	`CogniteAsset` / `CogniteTimeSeries`
`fileSchemaSpace` / `fileViewExternalId`	string	Optional	Schema space and view for the files that aliases update enriches.	`cdf_cdm` / `CogniteFile`
`timeseriesAliasPattern`, `assetAliasPattern`, `fileAliasPattern`	list of strings	Optional	Regex patterns that find the tag in an instance’s `name`. See Alias patterns.	Tag and document number patterns
`timeseriesAliasSelection`, `assetAliasSelection`, `fileAliasSelection`	string	Optional	What to keep when several patterns match one name: `all` or `longest`.	`longest`, `all`, `all`

Aliases update runtime parameters

Edit extraction_pipelines/ctx_aliases_update.config.yaml to control which instances each run processes:

Parameter	Type	Required	Description	Default
`runAll`	boolean	Optional	Fetch all instances, not only those missing `aliases`.	`false`
`updateAll`	boolean	Optional	Rebuild generated aliases on every fetched instance while keeping hand-curated ones. Implies `runAll`.	`false`
`removeOldAliases`	boolean	Optional	Discard every existing alias and write only this run’s output. Implies `runAll`.	`false`

The fileView section is optional. Remove it from the config to leave file metadata untouched.

Alias patterns

Aliases update uses one list of regex patterns per view to find the tag inside an instance’s name, so time series, assets, and files can follow different naming conventions. The alias written back is the pattern’s capture groups joined by -. Separators inside a captured group (_, ., :) are also rewritten to -. With the default patterns, the time series name VAL_23-KA-9101:X.Value gets the alias 23-KA-9101, which is what makes matching on aliases work across differently formatted names. When several patterns match one name, the view’s AliasSelection setting decides what to keep: all writes one alias per matching pattern, and longest writes only the longest one. For example:

timeseriesAliasPattern:
  - '([0-9]{2}[-_.:][A-Z]{2,4}[-_.:][0-9]{4,5})'
  - '([A-Z]{2,4}[-_.:][0-9]{4,5})'
timeseriesAliasSelection: longest
assetAliasPattern:
  - '([A-Z]{2,4}[-_.:][0-9]{4,5})'
  - '([0-9]{2}[-_.:][A-Z]{2,4}[-_.:][0-9]{4,5})'
assetAliasSelection: all

A name that no pattern matches produces no alias. The function logs No alias extracted based on configured <view> regular expression for name: … for the first ten names, then a total. Many of these warnings usually point at the pattern, not the data. When nothing is extracted, aliases is written as null so a previous list is cleared. By default, fileAliasPattern matches document numbers, so PH-25578-P-4110006-001.pdf yields PH-25578-P-4110006-001 and PH-25578-P-4110006. Keep fileAliasSelection: all for these patterns, since longest would drop the shorter number. Files also get their file name without its final extension as an alias, but only when a pattern matches. AliasSelection never discards that file name alias. Follow these rules when you write your own patterns:

Use character classes, not backslash escapes. Write [0-9], not \d. The Cognite Toolkit substitutes variables as a regex replacement, and a backslash escape fails the build.
Accept - as a separator between groups. Generated aliases join groups with -, and the function recognizes its own earlier output by feeding a stored alias back through the pattern. A pattern that can’t match - never recognizes its own aliases, so updateAll leaves stale ones in place. Also accept _ so aliases written by earlier runs that used underscores are still recognized.

A write, and an updated count in the logs, happens only when the rebuilt alias list differs from what is stored.

Entity matching settings

These settings control which instances fn_dm_context_entity_matching reads, which properties it matches on, and how it approves and writes matches.

Entity matching variables

Variable	Type	Required	Description	Default
`targetViewExternalId`	string	Optional	External ID of the target view to match to.	`CogniteAsset`
`entityViewExternalId`	string	Optional	External ID of the entity view to match from.	`CogniteTimeSeries`
`targetViewSearchProperty`	string	Optional	Target property used to compute match candidates. See Search properties.	`name`
`entityViewSearchProperty`	string	Optional	Entity property used to compute match candidates. See Search properties.	`name`
`viewFilterProperty`	string	Optional	View property used to filter target and entity instances. See Filtering instances.	`tags`
`targetViewFilterValues`	list of strings	Optional	Values of `viewFilterProperty` that scope the target instances. Empty means no filter.	`[]`
`entityViewFilterValues`	list of strings	Optional	Values of `viewFilterProperty` that scope the entity instances. Empty means no filter.	`[]`
`primaryScopeProperty` / `secondaryScopeProperty`	string	Optional	Properties that restrict which targets each entity is matched against. See Scoped matching.	`''` / `''`

Entity matching runtime parameters

Edit extraction_pipelines/ctx_entity_matching.config.yaml to tune the matching run:

Parameter	Type	Required	Description	Default
`autoApprovalThreshold`	number	Optional	Matches with a confidence at or above this value are auto-approved and written to `contextualization_good`. Matches below it go to `contextualization_bad`.	`0.85`
`runAll`	boolean	Optional	Match all entities (`true`) or only those updated since the last run (`false`).	`false`
`dmUpdate`	boolean	Optional	Write approved matches as data model relationships (`true`) or to RAW only (`false`).	`true`
`removeOldLinks`	boolean	Optional	Remove existing target links before applying new matches.	`false`
`dataSetExternalId`	string	Optional	Data set for the function’s target cache and staged-match files. Remove it to write the files without a data set. A configured data set that doesn’t exist fails the run.	`ds_entity_matching`
`primaryScopeProperty` / `secondaryScopeProperty`	string	Optional	Set from the module variables of the same name.	`''`

If you change autoApprovalThreshold, clean up the relationships created by previous runs before re-running. Only matches created by this process are removed; manually created relationships are preserved.

Manual mapping and rule inputs

The submit stage reads two RAW tables in db_asset_entity_matching. The upload_data/ directory contains seed rows for both tables. Replace them with your own mappings and rules before you run cdf data upload. Manual mappings (contextualization_manual_input, uploaded from contextualization_manual_input.RawRows.csv) link a specific entity to a specific target:

Column	Description
`TsExternalId`	External ID of the entity (time series) to link.
`AssetExternalId`	External ID of the target (asset) to link to.
`TSName`	Optional. The entity’s name, for readability. The function doesn’t use it.
`Contextualized`	Leave empty. The function sets it to `true` after applying the mapping, and skips rows where it’s `true` unless `runAll` is `true`.

TsExternalId,AssetExternalId,TSName,Contextualized
pi:163655,45-TT-92506,45-TT-92506:X.Value,
pi:160665,23-PIC-96153,23-PIC-96153:Control Module:BA,

Manual mappings overwrite any existing mapping for the entity and take priority over rule and model matches. Rules (contextualization_rule_input, uploaded from contextualization_rule_input.RawRows.ndjson) link entities to targets whose names share a key. Each row holds a pair of regular expressions:

Column	Description
`EntityRegExp`	Regex applied to the entity’s `name`.
`AssetRegExp`	Regex applied to the target’s `name`.

For each rule, the key is the rule’s position plus the concatenated capture groups. An entity is linked to every target that gets the same key from the same rule. Rows missing either regex are skipped with a warning.

{"key": 1, "columns": {"AssetRegExp": "(\\d{2})[-_.:]([A-Z]{2})[-_.:](\\d{4,5})", "EntityRegExp": "(\\d{2})[-_.:]([A-Z]{2})[-_.:](\\d{4,5})"}}

With this rule, the time series VAL_23-KA-9101:X.Value and the asset 23-KA-9101 both capture 23, KA, and 9101, so they get the same key and the rule links them. Keep rules specific: a key that resolves to many targets gives one entity many links. Check the entity_rule_keys and asset_rule_keys columns in contextualization_good to see which keys produced a match.

Backslash escapes such as \d are fine in the rule table, because the rows are uploaded as data. The alias patterns in config.<env>.yaml are different: those must use character classes. See Alias patterns.

Search properties

targetViewSearchProperty and entityViewSearchProperty name the property whose value is passed to the matching model. The default is name. Set them to aliases when source system tags differ from display names, so matching compares the normalized tags that aliases update wrote. List-valued properties are reduced differently on each side:

Entities: only the longest usable entry is used, so one time series isn’t submitted as several match candidates. ["DB-9101", "23-DB-9101"] becomes 23-DB-9101. Values of equal length keep the first.
Targets: every usable entry is kept, so alternate spellings on an asset remain matchable.

A single string is used as-is. When the property holds nothing usable (it’s unset, an empty list, or a blank string), the instance falls back to matching on name. Empty entries in an otherwise populated list are dropped, so ["pi:1", ""] matches on pi:1 only.

If you configure a property name that doesn’t exist on the view, for example alias instead of aliases, every instance falls back to name. The run still produces matches, but not on the property you intended. Confirm the property name against your deployed view.

Filtering instances

viewFilterProperty is the view property used to filter target and entity instances, together with targetViewFilterValues and entityViewFilterValues. The filter applies only when the corresponding values list is non-empty. Only entity matching uses it; aliases update doesn’t filter. The default tags matches core data model views such as CogniteAsset. If your data model uses a different property, for example labels in the Capital Facilities Information Handover Specification (CFIHOS) oil and gas domain model, set it accordingly:

viewFilterProperty: labels
targetViewFilterValues: []
entityViewFilterValues: []

Scoped matching

primaryScopeProperty and secondaryScopeProperty are optional. Leave both empty (the default) to run one unscoped predict job against every target. Both property names must exist on the entity view and the target view. When primaryScopeProperty is set, the submit stage groups entities by their primary and secondary values and starts one predict job per group that has targets. Rule-based matching runs on the same groups. Manual mappings stay global, and the matching model fitted for the first job is reused by the others.

Entity scope	Matched against
Primary and secondary both empty or missing	Every target, in a job of its own.
Primary set, secondary empty or set	Targets with the same primary and the same secondary. Targets whose `tags` include `ScopeWideDetect` are candidates in every secondary scope of their primary scope, but never in another primary scope.
Non-empty scope with no matching targets	Skipped, and logged as `No assets in scope … source record(s) not matched`.

A missing property value is read as empty, so an entity with a primary value but no secondary value is its own scope, not the same as having no scope. To try scoped matching without an enterprise site/unit model, use the test resources in the module’s testing/ directory. Don’t leave the copied files in data_modeling/ or transformations/ unless you intend to deploy them.

Architecture

The pack deploys two functions into a dedicated function space, orchestrated by the wf_ctx_entity_matching workflow and a scheduled trigger. Each function reads its runtime parameters from an extraction pipeline config. RAW holds the manual and rule inputs, the good and bad match results, and the incremental state and job queue. CDF files in ds_entity_matching hold the target cache and staged matches.

Data flow

The workflow runs aliases update first to normalize tags, then the submit and collect stages of the matching function. Manual mappings and rule-based matches always take priority over model matches for the same entity.

Aliases update

fn_dm_context_aliases_update reads ep_ctx_aliases_update, pages through instances with only name and aliases, derives aliases with the view’s alias patterns, and writes each page back before reading the next. By default, only instances missing aliases are fetched. A failed read or write fails the function call, so matching doesn’t run on stale aliases.

Submit stage

The submit stage of fn_dm_context_entity_matching:

Reads the ep_ctx_entity_matching config and the manual and rule mapping tables.
Reads the target instances through the data modeling sync endpoint and caches them in a CDF file, em_target_cache_<key>.json. Later runs merge only the sync changes into the same file instead of paging the whole view again.
Reads the entities not yet matched (or all entities when runAll is true), one page at a time with only the properties matching uses.
Applies manual mappings, which overwrite any existing mapping, then rule-based matching with the configured regex patterns.
Starts the entity matching predict job for the remaining candidates without waiting for it, one job per scope when scoped matching is configured.
Stages the manual and rule matches in a CDF file, em_staged_matches_<jobId>.json, and adds the job to the queue in contextualization_state_store.

If the target view, spaces, or filter change, the cache key changes and a new cache file and state row are created. The old ones are left unused.

Collect stage

The collect stage polls queued predict jobs in parallel, with up to 10 workers, every 30 seconds, for at most 7 minutes per run. When a job finishes and time remains, the next queued job is started. Finished results are merged with the staged matches, split by autoApprovalThreshold into the good and bad tables, written to the data model when dmUpdate is true, and removed from the queue. Jobs still running are picked up by the next collect run. The function stores a digest of each cache and staged-match file in the RAW state row and checks file contents against that digest. A principal that can overwrite files, but not that RAW table, can’t inject content.

Incremental processing and state

Key	Written by	Meaning
`state_predict_job_<jobId>` (RAW)	Submit	A predict job waiting to be collected.
`state_target_sync_<key>` (RAW)	Submit	Sync cursor, cache file ID, page size, and target count.
`em_staged_matches_<jobId>.json` (CDF file)	Submit	Manual and rule matches staged for that job.
`em_target_cache_<key>.json` (CDF file)	Submit	Cached targets. Overwritten when targets change, never deleted.

With runAll: false (the shipped default), only entities updated since the last run are processed. Set runAll: true or clear the state table to reprocess everything.

Resilience

Every task fails its function call on error, so the workflow stops instead of matching on stale aliases or partial input. A failed read of the time series, the manual or rule mappings, or a staged-match file fails the run. A job whose staged matches can’t be read stays queued for the next collect run. Aliases update and collect retry up to 3 times. Submit isn’t retried, because a retry after a partial run could queue a second predict job for the same entities; the next workflow run picks those entities up again.

Access

The gp_entity_matching_processing group grants the workflow’s service principal these access control lists (ACLs):

rawAcl read and write on the db_asset_entity_matching database.
dataModelsAcl read and dataModelInstancesAcl read and write.
filesAcl read and write scoped to the ds_entity_matching data set, for the target cache and staged-match files, and datasetsAcl read to resolve the data set.
entitymatchingAcl, functionsAcl, sessionsAcl, extractionConfigsAcl, and extractionRunsAcl for running the functions and the matching jobs.

Troubleshooting

Problems are grouped by where they show up. Check the function logs in Build Solutions > Functions first: both functions log the configuration they read and the counts they processed.

Deployment and access problems

cdf deploy fails with Insufficient permissions

Cause: The service account used by the Cognite Toolkit is missing required CDF capabilities.Resolution: Grant capabilities to deploy functions, workflows, extraction pipelines, RAW, data sets, spaces, and groups.

cdf data upload fails with data plugin not enabled

Cause: [plugins] data = true is missing from cdf.toml.Resolution: Add data = true under [plugins] in cdf.toml and retry the upload command.

The workflow fails with an access error

Cause: The workflow credentials or the processing group are misconfigured.Resolution: Verify that workflowClientId, workflowClientSecret, and entity_matching_processing_group_source_id in config.<env>.yaml point to the correct IdP application and group source ID. See Access management.

Submit fails on the target cache or with a data set error

Cause: The processing group lacks filesAcl on ds_entity_matching, or dataSetExternalId names a data set that doesn’t exist.Resolution: Confirm that the data set exists and that the processing group has filesAcl read and write on it and datasetsAcl read. Alternatively, remove dataSetExternalId from the extraction pipeline config.

The workflow never runs on its own

Cause: The default workflowSchedule only fires on 29 February.Resolution: Set workflowSchedule to a cron expression, for example '0 2 * * *', and redeploy.

Alias problems

cdf build fails on an alias pattern

Cause: The pattern uses a backslash escape such as \d.Resolution: Use character classes such as [0-9] instead. See Alias patterns.

Many No alias extracted warnings in the logs

Cause: The alias patterns don’t match your naming convention.Resolution: Adjust timeseriesAliasPattern, assetAliasPattern, or fileAliasPattern to match your names.

updateAll leaves stale aliases in place

Cause: The alias pattern doesn’t accept - as a separator, so the function can’t recognize its own earlier aliases.Resolution: Add - (and _) to the separators the pattern accepts.

Matching problems

No matches created after the workflow run

Work through these causes in order:

Predict jobs are still queued. Run the workflow again. The collect stage picks up jobs that were still running when the previous run ended.
Input tables are missing. Verify that cdf data upload populated contextualization_manual_input and contextualization_rule_input.
Search properties don’t exist on the views. Confirm that targetViewSearchProperty and entityViewSearchProperty match real property names. See Search properties.
The configuration doesn’t match your data. Check that filter values, scope properties, views, spaces, and versions in the extraction pipeline config match your data in CDF. Clear targetViewFilterValues and entityViewFilterValues to remove the filter.

Low match rate despite correct configuration

Cause: Aliases are missing, or matching runs on name instead of aliases.Resolution: Check the fn_dm_context_aliases_update logs, and set the search properties to aliases if your tags differ from display names.

Property does not exist in view (400 error)

Cause: viewFilterProperty names a property the views don’t have. The error appears only once targetViewFilterValues or entityViewFilterValues is non-empty, so a wrong name can go unnoticed until you start filtering.Resolution: Set viewFilterProperty to the property your views use, for example tags for core data model views or labels for CFIHOS. See Filtering instances.

Warning that external IDs exist in more than one instance space

Cause: Several configured spaces reuse external IDs.Resolution: Each instance is still matched in its own space. If the warning names assets, confirm that links point to the expected space. Manual mappings apply to every copy of an external ID. See Multiple instance spaces.

Matches are written to RAW but not to the data model

Cause: dmUpdate is set to false.Resolution: Set dmUpdate: true in extraction_pipelines/ctx_entity_matching.config.yaml and re-run.

Duplicate or stale relationships after a threshold change

Cause: autoApprovalThreshold changed without cleaning up prior matches.Resolution: Remove relationships created by previous runs, or set removeOldLinks: true, before re-running. Manually created relationships are preserved.

The workflow processes all entities on every run

Cause: runAll is true, or the state table was cleared.Resolution: Set runAll: false for incremental runs. Confirm that contextualization_state_store exists and is being updated.

For issues not covered here, contact Cognite support.

Support

Field	Details
Team	Cognite Value Delivery Team
Documentation	Cognite product documentation
Questions and feedback	Deployment Packs group on Cognite Hub
Version	1.0.0
Last updated	2026

Configure and deploy

CDF Entity Matching reference

Who it’s for

What’s included

Prerequisites

Install and deploy

Verify the deployment

Configuration reference

General variables

Multiple instance spaces

Function input

Aliases update settings

Aliases update variables

Aliases update runtime parameters

Alias patterns

Entity matching settings

Entity matching variables

Entity matching runtime parameters

Manual mapping and rule inputs

Search properties

Filtering instances

Scoped matching

Architecture

Data flow

Aliases update

Submit stage

Collect stage

Incremental processing and state

Resilience

Access

Troubleshooting

Deployment and access problems

Alias problems

Matching problems

Further reading

Support

​Who it’s for

​What’s included

​Prerequisites

​Install and deploy

​Verify the deployment

​Configuration reference

​General variables

​Multiple instance spaces

​Function input

​Aliases update settings

​Aliases update variables

​Aliases update runtime parameters

​Alias patterns

​Entity matching settings

​Entity matching variables

​Entity matching runtime parameters

​Manual mapping and rule inputs

​Search properties

​Filtering instances

​Scoped matching

​Architecture

​Data flow

​Aliases update

​Submit stage

​Collect stage

​Incremental processing and state

​Resilience

​Access

​Troubleshooting

​Deployment and access problems

​Alias problems

​Matching problems

​Further reading

​Support

Who it’s for

What’s included

Prerequisites

Install and deploy

Verify the deployment

Configuration reference

General variables

Multiple instance spaces

Function input

Aliases update settings

Aliases update variables

Aliases update runtime parameters

Alias patterns

Entity matching settings

Entity matching variables

Entity matching runtime parameters

Manual mapping and rule inputs

Search properties

Filtering instances

Scoped matching

Architecture

Data flow

Aliases update

Submit stage

Collect stage

Incremental processing and state

Resilience

Access

Troubleshooting

Deployment and access problems

Alias problems

Matching problems

Further reading

Support