> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.
# P&ID Annotation reference
> Automate P&ID diagram annotation in CDF with a workflow that tags assets and files, detects tags in diagrams, and links them to assets and files as data model annotations.
The **P\&ID Annotation deployment pack** (`dp:contextualization:cdf_p_and_id_annotation`) automates the annotation of Piping and Instrumentation Diagrams (P\&IDs) in Cognite Data Fusion. It deploys as a Cognite Toolkit module and uses a CDF Workflow to run two tagging transformations followed by a CDF Function that detects tags in each diagram and links them to assets and files as `CogniteDiagramAnnotation` instances in the data model.
The pack ships production-tested code drawn from real customer implementations, with incremental processing via the Data Modeling Sync API, threshold-based approval and suggestion, and batch-level resilience for large P\&ID volumes.
P\&ID Annotation is a cherry-pickable module (`dp:contextualization:cdf_p_and_id_annotation`) inside the **Contextualization** deployment pack (`dp:contextualization`). You can add it on its own or alongside the other contextualization modules.
## Who it's for
**Target personas:**
* **Data engineers** — deploy, configure, and tune the annotation pipeline.
* **Subject matter experts** — engineers who validate P\&ID-to-asset annotations.
**Use this pack when:**
* You need to contextualize P\&ID documents by linking them to assets and other files in your CDF data model.
* You want a production-ready annotation pipeline without building one from scratch.
* Your project ingests P\&ID files from sources such as SharePoint and needs to match them to asset hierarchies from systems such as SAP.
* You need incremental processing — only new or updated files should be annotated on each run.
**When not to use this pack:**
* If your P\&ID documents require a fundamentally different annotation engine — this pack covers standard diagram detection and CDF annotation.
* If P\&ID files have not yet been ingested into the CDF file store — the pack requires files to be present before it can annotate them.
* If you need to match time series to assets rather than annotate documents, use the [CDF Entity Matching](/cdf/deploy/cdf_toolkit/references/packages/entity_matching) deployment pack instead.
## What's included
| Component | Description | Optional? |
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| Asset tagging transformation (`asset_tagging_tr`) | Adds a `PID` tag to qualifying assets to control which assets enter the annotation matching process. Provided as a customizable example. | No |
| File tagging transformation (`file_tagging_tr`) | Adds a `PID` tag to qualifying files to control which files enter the annotation process. Provided as a customizable example. | No |
| Annotation function (`fn_dm_context_files_annotation`) | Reads new or updated files via the Data Modeling Sync API, detects tags in each P\&ID, matches them to assets and files, applies threshold logic, and writes `CogniteDiagramAnnotation` instances. | No |
| Workflow | Runs the asset and file tagging transformations, then the annotation function (`dependsOn` both), each with 3 retries and `abortWorkflow` on failure. | No |
| Scheduled trigger | Cron trigger that runs the workflow daily at 04:00 UTC using the configured credentials. | Yes |
| Extraction pipeline (`ep_ctx_files_pandid_annotation`) | Holds the runtime parameters, view configuration, and thresholds the annotation function reads. | No |
| RAW database and tables | `documents_tags` (document→tag), `documents_docs` (document→document), and `files_state_store` (incremental cursor). | No |
| Access group and dataset | A `files.processing.groups` access group bound to a configured IdP group source ID, and a files dataset for lineage. | No |
| Location filter | An example location filter for scoping data by location. | Yes |
## Prerequisites
Verify all of the following before you start:
* Cognite Toolkit version **0.8 or later** is installed. See [Setting up](/cdf/deploy/cdf_toolkit/guides/setup).
* A `cdf.toml` file exists in your project root. If it is missing, run `cdf modules init` and select **Create toml file (required)**.
* Authentication is configured and verified — run `cdf auth init` and `cdf auth verify` using a local `.env` file. See [Authentication and authorization](/cdf/deploy/cdf_toolkit/guides/auth).
* P\&ID files are already ingested into the CDF file store, and an asset hierarchy exists in the data model for matching.
* You have an IdP application for the function and scheduled workflow (`IDP_CLIENT_ID` / `IDP_CLIENT_SECRET`) and the IdP group source ID for the `files.processing.groups` access group.
## Install and deploy
You can add the P\&ID Annotation module to an existing Toolkit project or start a new project from scratch.
Add the module from your project directory:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules add -d cdf_p_and_id_annotation
```
1. In an empty directory, initialize a project:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules init .
```
2. In the interactive menu, choose **Contextualization**, use **Space** to select **Contextualization P\&ID Annotation**, then press **Enter** to confirm.
The Toolkit creates the module under `modules/contextualization/cdf_p_and_id_annotation/` and updates your `config..yaml` files. Before building, open `config.dev.yaml` and set at minimum `project`, `organization`, `location_name`, `source_name`, the schema and instance spaces, and the credentials (`functionClientId` / `functionClientSecret` and `files_location_processing_group_source_id`) — see [Configuration reference](#configuration-reference). Then build and deploy:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf build
```
The Toolkit substitutes template variables and writes artifacts to the `build/` directory. If you need to change the project name or any variables marked ``, edit `config..yaml` first — see [Configure, build, and deploy modules](/cdf/deploy/cdf_toolkit/guides/usage).
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy --dry-run
```
Inspect the output and confirm that configurations look correct before deploying.
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy
```
The Toolkit deploys only configurations that have changed since the last run.
For governed production deployments, automate build, dry-run, and deploy in a pipeline. See [Set up CI/CD pipelines](/cdf/deploy/cdf_toolkit/guides/cicd/index).
After deployment, your project should include:
```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
modules/
└── contextualization/
└── cdf_p_and_id_annotation/
```
## Verify the deployment
After deploying, confirm success with these checks:
1. The `cdf deploy` command finishes without errors and reports successful deployment of Functions, Workflows, Transformations, Extraction Pipelines, RAW, the data set, and the access group.
2. In CDF, navigate to **Integrate > Data Workflows** and verify that the annotation workflow is listed (external ID `p_and_id_annotation` by default).
3. Trigger the workflow with **Run** (or wait for the daily trigger). Confirm that both tagging transformations complete before the annotation function, and that all tasks finish without errors.
4. Navigate to **Build Solutions > Functions**, open `fn_dm_context_files_annotation`, select **View logs**, and verify that files were processed without errors.
5. Navigate to **Integrate > Staging (RAW)** and verify that `documents_tags`, `documents_docs`, and `files_state_store` have been created and populated.
6. Use CDF Search in Industrial Tools to confirm annotations on P\&ID documents — purple boxes link to assets, orange boxes link to files.
The deployment is successful when the workflow completes without errors, the RAW match tables are populated, and annotations appear on the P\&ID documents.
## Configuration reference
Module variables are set per environment in `config..yaml`. Update these before running `cdf build`.
Each variable takes its default unless you override it. Where the default is `—`, a placeholder (for example `YourOrg`, `LOC`, `SOURCE`, ``), or a `${...}` value read from `.env`, you must provide your own value before deploying.
| Variable | Description | Default |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| `project` | CDF project external ID to deploy to. | — |
| `organization` | Organization prefix used in the target view external IDs (for example, `YourOrgFile`, `YourOrgAsset`). | `YourOrg` |
| `location_name` | Location or asset name used in resource naming (replaces `LOC`). | `LOC` |
| `source_name` | Source system the P\&ID files come from (replaces `SOURCE`). | `SOURCE` |
| `files_dataset` | Data set for lineage and the function code bundle. | `ds_files_LOC` |
| `schemaSpace` | Schema space containing the file and asset views. | `sp_enterprise_process_industry` |
| `annotationSchemaSpace` | Schema space for the annotation view. | `cdf_cdm` |
| `viewVersion` | Version of the file and asset views. | `v1.0` |
| `fileInstanceSpace` | Instance space for file instances. | `springfield_instances` |
| `assetInstanceSpace` | Instance space for asset instances. | `springfield_instances` |
| `functionClientId` / `functionClientSecret` | IdP credentials used by the function and scheduled trigger. | `${IDP_CLIENT_ID}` / `${IDP_CLIENT_SECRET}` |
| `workflow` | Workflow external ID (also used by the scheduled trigger). | `p_and_id_annotation` |
| `files_location_processing_group_source_id` | IdP group source ID for the `files.processing.groups` access group. | `` |
### Runtime parameters (extraction pipeline)
Annotation behavior is controlled by `extraction_pipelines/ctx_files-pandid_annotation.config.yaml` (`ep_ctx_files_pandid_annotation`), not `config..yaml`.
| Parameter | Description | Default |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `autoApprovalThreshold` | Confidence at or above this value creates an approved annotation. | `0.85` |
| `autoSuggestThreshold` | Confidence at or above this value creates a suggested annotation. | `0.50` |
| `runAll` | Annotate all files (`true`) or only files updated since the last run (`false`). | `true` (shipped config) |
| `cleanOldAnnotations` | Delete annotations created by previous runs before annotating. | `false` |
| `debug` / `debugFile` | Process a single named file with verbose logging. | `false` |
| `rawDb`, `rawTableState`, `rawTableDocTag`, `rawTableDocDoc` | RAW database and tables for state and results. | `ds_files__`, `files_state_store`, `documents_tags`, `documents_docs` |
The `data` section configures the views to annotate — the file view (`type: diagrams.FileLink`) and one or more entity views such as the asset view (`type: diagrams.AssetLink`) — each with a `searchProperty` (default `name`) and an optional `filterProperty` / `filterValues` (for example, `tags: ["PID"]`) to scope matching.
If you change `autoApprovalThreshold` or `autoSuggestThreshold`, set `cleanOldAnnotations: true` for one run to remove the annotations created by previous runs before re-annotating. Only annotations created by this process are removed; manually created annotations are preserved.
## Architecture
The pack deploys a CDF Function, two tagging transformations, a workflow with a daily trigger, an extraction pipeline holding the runtime parameters, RAW tables, an access group, and a dataset.
### Data flow
The workflow tags qualifying assets and files, then runs the annotation function. The function detects tags in each P\&ID, matches them to assets and files, and creates annotations classified by the two confidence thresholds.
```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
flowchart TD
Trigger["Scheduled trigger · daily 04:00 UTC"] --> WF["Annotation workflow"]
WF --> AT["asset_tagging_tr tag assets with PID"]
WF --> FT["file_tagging_tr tag files with PID"]
AT --> Fn["fn_dm_context_files_annotation"]
FT --> Fn
Files["File view · P&ID documents new/updated via Sync API"] --> Fn
Assets["Asset view"] --> Fn
Fn --> Detect["Detect tags in diagram match to asset & file names"]
Detect --> Score{"match confidence"}
Score -->|≥ autoApprovalThreshold 0.85| Approve["Approved annotation"]
Score -->|≥ autoSuggestThreshold 0.50| Suggest["Suggested annotation"]
Approve --> DM["CogniteDiagramAnnotation diagrams.AssetLink · diagrams.FileLink"]
Suggest --> DM
Fn --> RawTag[("RAW · documents_tags (doc→tag)")]
Fn --> RawDoc[("RAW · documents_docs (doc→doc)")]
State[("RAW · files_state_store")] -.->|incremental cursor| Fn
Fn -.->|update cursor| State
```
### Annotation pipeline
The workflow runs in three stages:
1. **Tagging** — `asset_tagging_tr` and `file_tagging_tr` add a `PID` tag to qualifying assets and files. These are customizable examples that control which records the `filterProperty` / `filterValues` settings admit into matching.
2. **Detection and matching** — the annotation function reads new or updated files via the Sync API, detects tags in each diagram, and matches them against the configured file and asset views using each view's `searchProperty`.
3. **Annotation creation** — matches at or above `autoApprovalThreshold` are approved and those at or above `autoSuggestThreshold` are suggested; both are written as `CogniteDiagramAnnotation` instances (`diagrams.AssetLink` for assets, `diagrams.FileLink` for files). Document-to-tag and document-to-document matches are also written to the `documents_tags` and `documents_docs` RAW tables.
### Run modes
| Mode | Behavior |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| Incremental (`runAll: false`) | Processes only new or updated files using the `files_state_store` cursor; unchanged files are skipped. |
| ALL (`runAll: true`) | Clears cursor and per-run progress in RAW and reprocesses every matching P\&ID file. |
| Full cleanup (`cleanOldAnnotations: true`) | Also deletes annotations created by previous runs before reprocessing. Only annotations created by this process are removed. |
### Resilience
The annotation function processes files in batches with up to three retries per batch, then falls back to processing files individually. Failed individual files are logged and skipped without blocking the run. Annotation external IDs prevent duplicate annotations across runs when cleanup is disabled, and each successful batch persists its cursor so a failed run resumes without reprocessing.
## Troubleshooting
| Symptom | Likely cause | Resolution |
| -------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Workflow fails with an access error | The `files.processing.groups` group or function credentials are misconfigured. | Verify `functionClientId` / `functionClientSecret` and `files_location_processing_group_source_id` in `config..yaml` point at the correct IdP application and group source ID. See [Access management](/cdf/access). |
| No annotations created after the workflow run | Assets or files are missing the `PID` tag, or search properties do not match the data model. | Confirm `asset_tagging_tr` and `file_tagging_tr` completed, and that the `searchProperty` and `filterValues` in the extraction pipeline config align with your schema. |
| No annotations created after the workflow run | Filter criteria in the configuration do not match any data. | Check the extraction pipeline configuration and edit the filters to match your data, or remove the `filterProperty` and `filterValues` config lines. |
| No annotations created after the workflow run | The test data's views, schema space, instance space, or version do not match the configuration. | Check the extraction pipeline configuration and make sure it matches your data in CDF. |
| Duplicate annotations on re-run | `cleanOldAnnotations` is disabled and annotation state changed. | Set `cleanOldAnnotations: true` for one run, or ensure external ID generation is consistent across runs. |
| Workflow processes all files on every run | `runAll` is `true`, or the state table was cleared. | Set `runAll: false` for incremental runs and confirm `files_state_store` exists and is being updated. |
| Too many low-quality annotations | Thresholds are too low for your data. | Raise `autoApprovalThreshold` / `autoSuggestThreshold`, then run once with `cleanOldAnnotations: true` to remove prior annotations. |
| `cdf deploy` fails with "Insufficient permissions" | Toolkit service account is missing required CDF capabilities. | Grant capabilities to deploy Functions, Workflows, Transformations, Extraction Pipelines, RAW, Data Sets, and groups. |
For issues not covered here, contact Cognite support.
## Support
| Field | Details |
| ---------------------- | --------------------------------------------------------------------------- |
| Team | Cognite Value Delivery Team |
| Documentation | [Cognite product documentation](/) |
| Questions and feedback | [Deployment Packs group on Cognite Hub](https://hub.cognite.com/groups/578) |
| Version | 1.0.0 |
| Last updated | 2026 |