> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Cognite Data Fusion (CDF) is an industrial DataOps platform. Prefer English docs unless the user asks for another locale.
> Follow /_llms/ index links recursively until you reach page URLs ending in .md. Fetch those Markdown twins instead of HTML.
> For REST APIs, default to calendar version 20230101 (stable). Use 20230101-beta or 20230101-alpha only if the user is on a preview API. Cite paths under /api-reference/concepts/<version>/ and the Cdf-Version header. Prefer the Python SDK (cognite-sdk) unless the user is in JavaScript/TypeScript.
> Start here by task: platform overview /cdf/index.md; data modeling /cdf/dm/index.md; data integration /cdf/integration/index.md; access /cdf/access/index.md; CDF Toolkit /cdf/deploy/cdf_toolkit/index.md; REST quickstart /dev/quickstart.md; Python SDK https://cognite-sdk-python.readthedocs-hosted.com/en/stable/; Atlas AI /cdf/atlas_ai/index.md; plan an Atlas AI agent /cdf/atlas_ai/concepts/atlas_ai_agent_planning.md; Flows apps /cdf/flows/index.md; MCP and IDE setup /dev/guides/ide_ai_integration.md.
> For implementation work, also load /skill.md.

# Operate matching and annotation

> Step-by-step guide to scope, run incremental reruns, and maintain entity matching and diagram annotation pipelines without losing SME-confirmed links.

Treat contextualization as a repeatable pipeline, not a one-off project. After you [prepare aliases and match fields](https://docs.cognite.com/cdf/integration/guides/contextualization/prepare_aliases.md), you scope which sources and targets a run can see, apply deterministic links before machine learning, store what did not match, and rerun when data or mappings change.

This procedure is for data engineers who run matching in Cognite Data Fusion (CDF) or through the [CDF Entity Matching](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md) and [File Annotation](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/file_annotation.md) deployment packs. It does not replace the [CDF Entity Matching configuration reference](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#configuration-reference), [File Annotation configuration reference](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/file_annotation.md#configuration-reference), or deployment pack troubleshooting tables. Use those pages for YAML flags and error messages.

## Prerequisites

Before you start:

* Complete alias and match-field preparation on enterprise targets. See [Prepare aliases and match fields](https://docs.cognite.com/cdf/integration/guides/contextualization/prepare_aliases.md).
* Functional locations and sources live in the enterprise layer with stable identities. Functional locations are assets in the graph. See [About contextualization](https://docs.cognite.com/cdf/integration/concepts/contextualization/overview.md#what-must-exist-before-you-match).
* Decide whether you run matching in the CDF UI ([Match entities](https://docs.cognite.com/cdf/integration/guides/contextualization/match_entities.md)) or a Cognite Toolkit workflow.

> [!WARNING]
>
> Do not bulk-delete subject matter expert (SME)-confirmed matches, manual mapping tables, or regex rules to “start fresh.” Those artifacts are production data. Scope a rerun, adjust aliases, or reprocess only the entities that changed.

## Run and maintain pipelines

**1. Scope candidates by site, plant, or unit**

Limit each run by site, plant, or unit. Identical tags at different sites must not match each other. Aliases make the same tag text match across sites, and scope stops those false matches. Pick one scoping mechanism and use it consistently.

| Where you run | How to scope |
| - | - |
| CDF UI pipeline or quick match | Select **data sets** that contain only the sources and target assets for one site or unit. See [Match entities](https://docs.cognite.com/cdf/integration/guides/contextualization/match_entities.md). |
| Enterprise graph design | Isolate plants in separate [instance spaces](https://docs.cognite.com/cdf/dm/dm_concepts/dm_spaces_instances.md) or under distinct root `CogniteAsset` nodes in one hierarchy. See [Use one asset hierarchy](https://docs.cognite.com/cdf/integration/concepts/contextualization/overview.md#use-one-asset-hierarchy). |
| CDF Entity Matching deployment pack | Set `filterProperty` and `filterValues` on the entity view in the extraction pipeline config. The function then reads only the instances for that scope. See [CDF Entity Matching configuration reference](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#configuration-reference). |
| File Annotation deployment pack | Set `primaryScopeProperty` to the site or plant. Set `secondaryScopeProperty` when one site has several units. Launch then matches a file only to entities in the same scope, plus any entities tagged `ScopeWideDetect` in that primary scope. See [File Annotation scope](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/file_annotation.md#scope). |

> [!NOTE]
>
> A [location filter](https://docs.cognite.com/cdf/locations.md) scopes search and 3D in apps. It does not replace instance-space or data set scoping for matching pipelines.

**2. Apply deterministic links before machine learning**

Run the simplest approach that can create the relationship:

1. **Exact business key.** Write the link in a [transformation](https://docs.cognite.com/cdf/integration/concepts/transformation/index.md) when source and target share a stable key.
2. **Manual expert mappings.** Load curated source-to-target pairs (the CDF Entity Matching deployment pack stores these in `contextualization_manual_input`).
3. **Regex rules.** Encode repeatable patterns (the CDF Entity Matching deployment pack stores these in `contextualization_rule_input`; the CDF UI can save generated rules with a pipeline).
4. **Machine learning.** Use entity matching for the remainder.

The CDF Entity Matching deployment pack runs manual mappings, then rules, then ML. See [Architecture](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#architecture) in the deployment pack reference. Prefer adding a rule or mapping over retraining when the failure mode is a known pattern.

**3. Store and review unresolved matches**

Not every run produces a link. Plan for review instead of treating low coverage as a model-only problem.

* In the CDF UI, filter for **Unmatched entities** or **Different recommendation** when you validate a pipeline. See [Match entities](https://docs.cognite.com/cdf/integration/guides/contextualization/match_entities.md).
* In the CDF Entity Matching deployment pack, scores below `autoApprovalThreshold` land in `contextualization_bad`; approved matches land in `contextualization_good`. Review bad rows for missing targets, bad aliases, or scope that is too narrow or too wide.
* For diagram annotation, inspect files that returned no tag links before you rerun OCR or parsing. Low recall often means voided tags or missing `aliases` on assets, not a weak parser. If aliases look correct, see [Troubleshooting interactive diagrams](https://docs.cognite.com/cdf/integration/guides/contextualization/troubleshooting.md) for parser configuration.

Feed accepted fixes back into [Prepare aliases and match fields](https://docs.cognite.com/cdf/integration/guides/contextualization/prepare_aliases.md) or manual mapping tables. The next run must not repeat the same miss.

**4. Configure incremental reruns**

After the first pass, process only what changed unless you have a documented reason to reprocess everything.

**CDF UI**

* Save a **pipeline** tied to data sets. New ingested entities then enter the same model. When a data set receives new data, select **Rerun pipeline** on the entity matching overview. See [Rerun a matching model](https://docs.cognite.com/cdf/integration/guides/contextualization/match_entities.md#step-5-rerun-a-matching-model-in-existing-pipelines).
* Confirmed matches and saved rules persist with the pipeline. Reruns apply to new or unmatched entities according to the filters you selected.

**Cognite Toolkit deployment packs**

* Set `runAll: false`. The Entity Matching function then processes only entities updated since the last run. State is stored in `contextualization_state_store`. See [Incremental processing and state management](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#incremental-processing-and-state).
* For File Annotation, keep `cleanOldAnnotations` false on incremental runs so annotations from earlier runs stay. Set it to `true` only when you need to replace those annotations. See [Matching](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/file_annotation.md#matching).

Shipped deployment pack defaults often use `runAll: true` for first deployment. Switch to incremental mode after you validate the initial match rate.

**5. Decide when a full reprocess is justified**

Reprocess all entities or files only when the change affects matching logic globally, for example:

* You replaced the enterprise asset hierarchy or moved instance spaces.
* You changed `targetViewSearchProperty`, `entityViewSearchProperty`, or tokenizer-related fields across the project.
* You merged rename lists into `aliases` on a large share of targets and need to re-evaluate prior ML suggestions.

For a full reprocess in Cognite Toolkit deployment packs, set `runAll: true` or clear the relevant state table, then run the workflow. In the CDF UI, open the pipeline, adjust scope or model settings, and rerun on the full data set.

If you change `autoApprovalThreshold` in the CDF Entity Matching deployment pack, clean up relationships created by previous automated runs before you re-run. The deployment pack reference notes that manually created relationships are preserved. See [Runtime parameters](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#entity-matching-runtime-parameters).

**6. Trigger reruns when inputs change**

Schedule or trigger pipelines when any of these change:

| Change | Typical action |
| - | - |
| New time series or files ingested | Incremental deployment pack run or **Rerun pipeline** on the scoped data set. |
| New or updated `CogniteAsset` targets | Rerun matching to link sources to late-arriving functional locations. |
| Updated `aliases` or manual mappings | Rerun after [Prepare aliases](https://docs.cognite.com/cdf/integration/guides/contextualization/prepare_aliases.md); rules and mappings take effect on the next deployment pack run. |
| Source system rename project | Load authoritative alias mappings first, then scoped full reprocess for affected units. |

Use [CDF Workflows](https://docs.cognite.com/cdf/data_workflows/index.md) or scheduled triggers to run Cognite Toolkit modules after upstream transforms finish ingesting data.

## UI and Cognite Toolkit at a glance

If you are deciding where to run matching day to day, use this table to compare the CDF UI with Cognite Toolkit deployment packs before you return to the steps above for scoping and rerun rules.

| Concern | CDF UI | Cognite Toolkit deployment pack |
| - | - | - |
| Best for | Interactive review, quick match, SME validation | Scheduled production volume, RAW-backed rules and state |
| Incremental behavior | Rerun pipeline when data sets gain members | `runAll: false` plus state store tables |
| Expert mappings | Confirm matches in the UI; optional generated rules | CSV / NDJSON in `upload_data/`, loaded to RAW |
| Deep configuration | Match fields, data sets, thresholds in the wizard | Extraction pipeline YAML. See [CDF Entity Matching configuration reference](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/entity_matching.md#configuration-reference) and [File Annotation configuration reference](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/file_annotation.md#configuration-reference) |

## Next steps

* Read [Understanding contextualization quality](https://docs.cognite.com/cdf/integration/concepts/contextualization/quality.md), then deploy the [Contextualization Quality Dashboard](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/contextualization_status.md) deployment pack to score in-scope coverage and orphans.
* Return to [Prepare aliases and match fields](https://docs.cognite.com/cdf/integration/guides/contextualization/prepare_aliases.md) when bad rows point to tokenization or blocking issues.
* See [About contextualization](https://docs.cognite.com/cdf/integration/concepts/contextualization/overview.md) for tool choice and enterprise prerequisites.
