> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Atlas AI Property Extractor reference
> Extract structured property values from unstructured text in a CDF data model using an Atlas AI LLM agent, on a schedule, with add-new, append, and overwrite write modes.
The **Atlas AI Property Extractor** (`dp:atlas_ai:ai_extractor`) reads free text from a source property (for example, a maintenance notification description) and fills structured target properties (for example, `discipline`, `priority`, `category`) using an LLM agent deployed through Cognite Atlas AI. It deploys as a Cognite Toolkit module and runs a scheduled CDF Function that turns unstructured documentation into queryable data model properties without manual entry.
This module writes into an existing CDF view and supports per-property write modes — `add_new_only`, `append`, and `overwrite` — with epoch-based incremental processing.
## Who it's for
**Target personas:**
* **Data engineers** — deploy and configure the extractor and its schedule.
* **Subject matter experts** — domain experts who choose which properties to extract and validate results.
**Use this pack when:**
* You have documents, notifications, or free-text fields in CDF that contain technical properties (discipline, priority, category, tags) you want extracted and stored as structured data model properties.
* You want to enrich an existing CDF view with AI-generated summaries or derived metadata without building a custom extraction pipeline.
* You need to accumulate tags or keywords from multiple sources over time (append mode).
* You are building agentic solutions and need a structured data foundation for agentic reasoning and industrial search.
**When not to use this pack:**
* If your CDF project does not have Atlas AI enabled — this module requires an Atlas AI agent and cannot run without it.
* If you do not yet have a target CDF view — the module writes into an existing view and will not create one. Deploy your data model first.
* If you need rule-based or SQL-driven matching rather than LLM extraction — use the [CDF Entity Matching](/cdf/deploy/cdf_toolkit/references/packages/entity_matching) pack instead.
## What's included
| Component | Description | Optional? |
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| Atlas AI agent (`ai_property_extractor_agent`) | LLM agent that reads the source text and returns the extracted property values. | No |
| Extractor function (`fn_ai_property_extractor`) | CDF Function that queries source instances, calls the agent, writes results to the target view, and stamps the AI timestamp. Runs in a loop up to \~9 minutes per invocation. | No |
| Workflow (`wf_ai_property_extractor`) | Orchestrates the function, with a scheduled trigger (default daily at 04:00 UTC). Can also be triggered manually. | No |
| Extraction pipeline (`ep_ai_property_extractor`) | Holds the runtime configuration (views, properties, write modes, batching, prompt, state store) and tracks run status. | No |
| RAW state store | `ai_extractor_state` database with the `extraction_state` table — tracks the processing epoch and cursor for incremental runs. | No |
| Dataset (`ai_extractor`) | CDF dataset for lineage tracking. | No |
## Prerequisites
Verify all of the following before you start:
* Cognite Toolkit version **0.8 or later** is installed. See [Setting up](/cdf/deploy/cdf_toolkit/guides/setup).
* A `cdf.toml` file exists in your project root. If it is missing, run `cdf modules init` and select **Create toml file (required)**.
* Authentication is configured and verified — run `cdf auth init` and `cdf auth verify` using a local `.env` file. See [Authentication and authorization](/cdf/deploy/cdf_toolkit/guides/auth).
* Your CDF project has **Atlas AI Agents** enabled.
* A **target CDF view already exists** with a text property to read from and the properties you want to populate. For `append`/`overwrite`, the view also needs a `Timestamp` property for `aiTimestampProperty`.
* You have a service principal (app registration) in your identity provider (IdP) for the scheduled workflow to authenticate as. Have its client ID and client secret ready — these populate `IDP_CLIENT_ID` / `IDP_CLIENT_SECRET` in your `.env` (mapped to `workflowClientId` / `workflowClientSecret`).
## Install and deploy
You can add the module to an existing Toolkit project or start a new project from scratch.
The AI Property Extractor is the `cdf_ai_extractor` module of the **Atlas AI** deployment pack (`dp:atlas_ai`), which also ships out-of-the-box agents. Add this module from your project directory:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules add -d cdf_ai_extractor
```
1. In an empty directory, initialize a project:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf modules init .
```
2. In the interactive menu, choose **Atlas AI Deployment Pack**, use **Space** to select **AI Property Extractor Function**, then press **Enter** to confirm.
The Toolkit creates the module under `modules/solutions/cdf_ai_extractor/` and updates your `config..yaml` files. Before building, open `config.dev.yaml` and set at minimum the source view (`viewSpace`, `viewExternalId`, `viewVersion`), the `textProperty`, the properties to extract, the credentials (`workflowClientId` / `workflowClientSecret`), and — if using `append`/`overwrite` — the `aiTimestampProperty`. See [Configuration reference](#configuration-reference). Then build and deploy:
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf build
```
The Toolkit substitutes template variables and writes artifacts to the `build/` directory. If you need to change the project name or any variables marked ``, edit `config..yaml` first — see [Configure, build, and deploy modules](/cdf/deploy/cdf_toolkit/guides/usage).
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy --dry-run
```
Inspect the output and confirm that configurations look correct before deploying.
```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cdf deploy
```
The Toolkit deploys only configurations that have changed since the last run.
For governed production deployments, automate build, dry-run, and deploy in a pipeline. See [Set up CI/CD pipelines](/cdf/deploy/cdf_toolkit/guides/cicd/index).
After deployment, your project should include:
```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
modules/
└── solutions/
└── cdf_ai_extractor/
```
## Verify the deployment
After deploying, confirm success with these checks:
1. The `cdf deploy` command finishes without errors and reports successful deployment of the agent, function, workflow, extraction pipeline, dataset, and RAW resources.
2. In CDF, navigate to **Build Solutions > Functions** and confirm `fn_ai_property_extractor` is listed with a **Ready** status.
3. Navigate to **Integrate > Data Workflows** and confirm `wf_ai_property_extractor` is registered and its schedule is active.
4. Trigger the workflow (or call the function directly with `logLevel: DEBUG`) to run an initial extraction, and monitor it to completion.
5. Open the function logs and confirm that instances were processed and properties were written without errors.
6. Open the target view in **Data Modeling** and verify that the extracted properties appear on the expected instances.
7. In **Integrate > Extraction pipelines**, confirm `ep_ai_property_extractor` reports a successful run.
The deployment is successful when the function is ready, the workflow runs without errors, and extracted properties appear in the target view.
## Configuration reference
Module variables are set per environment in `config..yaml`. Update these before running `cdf build`. Keep client IDs and secrets in your `.env` file, not hardcoded.
Each variable takes its default unless you override it. Where the default is `—`, a placeholder, or a `${...}` value read from `.env`, you must provide your own value before deploying.
| Variable | Description | Default |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `project` | CDF project external ID to deploy to. | — |
| `viewSpace` / `viewExternalId` / `viewVersion` | Source view containing the instances to read from. | `cdf_cdm` / `CogniteAsset` / `v1` |
| `textProperty` | Property on the source view containing the text to extract from. | `description` |
| `propertiesToExtract` | JSON array of property IDs to extract. Empty extracts all non-filled properties. | `[]` |
| `targetViewSpace` / `targetViewExternalId` / `targetViewVersion` | Target view for writing, if different from the source view. | — / — / `v1` |
| `aiTimestampProperty` | `Timestamp` property on the target view used to track processed instances. **Required for `append` / `overwrite`.** | `""` |
| `aiPropertyMapping` | JSON object mapping source properties to different target properties. | `{}` |
| `processingFilters` | JSON array of extra data model filters for instance selection. | `[]` |
| `customPromptInstructions` | Extra instructions appended to the LLM prompt. | `""` |
| `batchSize` | Instances per query batch (1–100). | `10` |
| `llmBatchSize` | Instances per LLM call (1–50). | `1` |
| `functionSpace` | Space for the function deployment. | — |
| `agentExternalId` | Atlas AI agent external ID. | `ai_property_extractor_agent` |
| `extractionPipelineExternalId` | Extraction pipeline external ID. | `ep_ai_property_extractor` |
| `stateStoreEnabled` / `stateStoreDatabase` / `stateStoreTable` | State tracking toggle and RAW location. | `true` / `ai_extractor_state` / `extraction_state` |
| `stateStoreConfigVersion` | Bump to trigger a full re-run (new epoch). | `v1` |
| `workflow` | Workflow external ID. | `wf_ai_property_extractor` |
| `scheduleExpression` | Cron schedule for the workflow. | `0 4 * * *` |
| `workflowClientId` / `workflowClientSecret` | IdP credentials for the scheduled workflow. | `${IDP_CLIENT_ID}` / `${IDP_CLIENT_SECRET}` |
### Write modes
Each property can have its own write mode, configured in the extraction pipeline config's per-property `properties` list.
| Mode | Behavior | AI timestamp required? | Use case |
| ------------------------ | --------------------------------------------------------- | ---------------------- | ----------------------------------- |
| `add_new_only` (default) | Writes only when the target property is empty. | No | One-time enrichment. |
| `append` | Merges new values into existing lists and deduplicates. | Yes | Accumulating tags over time. |
| `overwrite` | Always replaces the target value with the new extraction. | Yes | Re-extraction after an LLM upgrade. |
## Architecture
The module deploys an Atlas AI agent, a CDF Function, a scheduled workflow, an extraction pipeline holding the runtime config, a RAW state store, and a dataset.
### Data flow
On each scheduled or manual run, the function loops — querying a batch, calling the agent, writing results, and stamping the timestamp — until all matching instances are processed or \~9 minutes elapse. The state store lets the next run resume where the previous one stopped.
```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
flowchart TD
Trigger["Scheduled trigger · daily 04:00 UTC"] --> WF["wf_ai_property_extractor"]
WF --> Fn["fn_ai_property_extractor
loops up to ~9 min per run"]
Src["Source view · textProperty"] --> Fn
Fn --> Q["Query batch
add_new_only: target empty · append/overwrite: AI timestamp < epoch"]
Q --> Agent["Atlas AI agent
ai_property_extractor_agent (LLM)"]
Agent --> Write["Write extracted properties
to target view"]
Write --> Stamp["Stamp aiTimestampProperty
(append / overwrite)"]
Stamp --> Q
Write --> Target["Target view
structured properties"]
State[("RAW · ai_extractor_state / extraction_state
epoch_start · cursor")] -.->|resume| Fn
Fn -.->|update| State
```
### Extraction loop
1. **Query a batch** — select instances from the source view where the text property exists and, depending on write mode, the target is empty (`add_new_only`) or the AI timestamp predates the current epoch (`append` / `overwrite`).
2. **Invoke the agent** — send the text to `ai_property_extractor_agent`, which returns the extracted property values (optionally several instances per call via `llmBatchSize`).
3. **Write to the target view** — upsert the values into the target view, following each property's write mode.
4. **Stamp and repeat** — set `aiTimestampProperty` on processed instances (append/overwrite), update the state cursor, and continue until done or the time budget is reached.
### State and re-runs
State is a single RAW row in `ai_extractor_state/extraction_state` holding `epoch_start`, `cursor`, and `config_version`. To reprocess everything, call the function with `resetState: true`, bump `stateStoreConfigVersion`, or disable the state store to process all matching instances every run. Manual runs accept `ExtractionPipelineExtId`, `logLevel`, and `resetState` as input parameters.
## Troubleshooting
| Symptom | Likely cause | Resolution |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Function shows a failed status after deployment | Atlas AI Agents not enabled, or the Toolkit account is missing capabilities. | Confirm Atlas AI Agents are enabled, and that `functions:write` and `workflows:write` are granted to the Toolkit CDF group. |
| `"Agent not found"` | The `ai_property_extractor_agent` agent is not deployed or accessible. | Verify the agent deployed successfully and the project has Atlas AI enabled. |
| `"View not found"` | `viewSpace` / `viewExternalId` / `viewVersion` are incorrect. | Confirm the external IDs in `config..yaml` match a view in your project. |
| Workflow runs but no properties are written | Source/target view misconfigured, text property empty, or targets already filled (`add_new_only`). | Check the function logs; confirm the views, `textProperty`, and target properties, and that instances have text to read. |
| Same records re-extracted on every run | `aiTimestampProperty` is missing or does not match a `Timestamp` property on the target view. | Add `aiTimestampProperty` to the config and create it as a `Timestamp` property on the target view. |
| `"aiTimestampProperty is required"` | A property uses `append` or `overwrite` without a configured timestamp. | Configure `aiTimestampProperty` and add the `Timestamp` property to the target view. |
| Values never refreshed after an LLM upgrade | Running in `add_new_only`/`append` when a full refresh is needed. | Use `overwrite` for the affected properties, or trigger a full re-run with `resetState: true` or by bumping `stateStoreConfigVersion`. |
| Workflow schedule never fires | Invalid cron (for example, February 29 in a non-leap year). | Update `scheduleExpression` to a valid recurring cron expression. |
| Function timeout | Batches too large or prompt too heavy. | Reduce `batchSize`, simplify `customPromptInstructions`, or raise `llmBatchSize`. |
For issues not covered here, contact Cognite support.
## Support
| Field | Details |
| ---------------------- | --------------------------------------------------------------------------- |
| Team | Cognite Value Delivery Team |
| Documentation | [Cognite product documentation](/) |
| Questions and feedback | [Deployment Packs group on Cognite Hub](https://hub.cognite.com/groups/578) |
| Version | 1.0.0 |
| Last updated | 2026 |