> ## Agent Instructions > Cognite Data Fusion (CDF) is an industrial DataOps platform. Prefer English docs unless the user asks for another locale. > Follow /_llms/ index links recursively until you reach page URLs ending in .md. Fetch those Markdown twins instead of HTML. > For REST APIs, default to calendar version 20230101 (stable). Use 20230101-beta or 20230101-alpha only if the user is on a preview API. Cite paths under /api-reference/concepts// and the Cdf-Version header. Prefer the Python SDK (cognite-sdk) unless the user is in JavaScript/TypeScript. > Start here by task: platform overview /cdf/index.md; data modeling /cdf/dm/index.md; data integration /cdf/integration/index.md; access /cdf/access/index.md; CDF Toolkit /cdf/deploy/cdf_toolkit/index.md; REST quickstart /dev/quickstart.md; Python SDK https://cognite-sdk-python.readthedocs-hosted.com/en/stable/; Atlas AI /cdf/atlas_ai/concepts/index.md; Flows apps /cdf/flows/index.md; MCP and IDE setup /dev/guides/ide_ai_integration.md. > For implementation work, also load /skill.md. # About data engineering Source: https://docs.cognite.com/cdf/data_engineering/index Model, integrate, transform, and consolidate data into structures that are suitable for building analytics solutions. Find, verify, visualize, and learn more about your data using Data explorer and Data catalog. Ensure data quality and integrity through secure access management and data set tracking. Build expressive and scalable data models to create a structured knowledge graph. Automate interdependent processes with built-in retry logic and error handling. Connect external systems and transfer data into CDF using extractors and connectors. Stream or batch-extract data into the CDF staging area for quality checks and cleanup. Move and enrich data from source systems into CDF's data model or custom models. Connect and enrich industrial data by matching entities and building relationships. # About data sets Source: https://docs.cognite.com/cdf/data_governance/concepts/datasets/index Data sets let you document and track data lineage, ensure data integrity, and allow 3rd parties to write their insights to your Cognite Data Fusion (CDF) project securely. ## What is a data set? Data sets group and track data by **its source**. For example, a data set can contain all work orders originating from SAP or the output data from a 3rd party partner's machine-learning model. Typically, organizations have one data set for each data ingestion pipeline in CDF. Examples of data sets showing SAP work orders, PI time series, and machine learning model outputs organized by their source A **data set** is a container for data objects with metadata about the data it contains. For example, you can use the data set metadata to document who is responsible for the data, upload documentation files, and describe the data lineage. In CDF, data sets are a separate resource type with a `/datasets` API endpoint. To define which **data objects**, such as events, files, and time series, belong to a data set, you specify the relevant `dataSetId` field for each data object. This is typically done programmatically in the data ingestion pipelines. Data objects can belong to only one data set so you can unambiguously trace the data lineage for each data object. You can organize the following resource types into data sets: * Assets * Events * Files * Time series * Sequences ## Why use data sets? For proper data governance, you need to trace the **data lineage** to understand where data originates from and be confident that the data is reliable. Data managers need to ensure **data integrity** and let **3rd parties write data** to CDF. ### Trace data lineage * **IT managers** need to know which data is currently available in their CDF project. * **Data scientists** need to know if they can rely on the input data for the use cases they're solving and who to contact if they need more information. * **IT support staff** need to know how the data is integrated to help troubleshoot any issues. ### Ensure data integrity * When **data engineers** and **IT managers** have designed, implemented, and approved the data ingestion pipelines, they need to protect the pipelines from accidental changes to keep the data valid and accurate. ### Let 3rd parties write data securely back to CDF * When you want **3rd parties** to write their insights back to your CDF project, you need to provide a safe container to hold their data and, at the same time, safeguard data from your other data pipelines. ## Data set concepts ### Write-protection We recommend that you **write-protect** data sets that contain production-critical data to preserve the integrity of the data. Only members of groups you explicitly grant access can write data to a write-protected data set. Other users, apps, or services can not change the data in the data set even if they have the necessary access rights to change the data objects contained in the data set. **To write-protect a data set:** [Create](/cdf/data_governance/guides/datasets/create_data_sets) or [edit](/cdf/data_governance/guides/datasets/edit_explore_data_sets) a data set and mark it as write protected. In [access management](/cdf/access/entra/guides/create_groups_oidc), use the **datasets** capability to set the **owner** action for a group and **scope** it to the `dataSetId`. | Desired access | Capability | Action | Scope | | - | - | - | - | | Write to a specific write-protected data set | datasets | owner | dataSetId | ### Governance status Set the governance status to define if the data set is **Governed** or **Ungoverned**. * **Governed** data sets have a designated owner and follow the governance processes for data in your organization. * **Ungoverned** data sets indicate to users that the data may not be reliable. If you want to use data from an ungoverned data set, we recommend contacting the owner or creator of the data set to learn more about the data. ### Labels Use labels to make data sets more discoverable and to group similar data sets. For example, you can use labels to specify the type of data it contains, the geographical region covered by the data, or which business unit uses the data. You can search for data sets by their labels. ### Archiving You can archive data sets to hide them from the user interface and make the data set unavailable to users. The data set and the data it contains won't be deleted, and you can always restore archived data sets later. ### Lineage Data lineage helps you be confident that the data in a data set is reliable. On the **lineage tab** for each data set, you can see who has created a data set, where the data comes from, and how the data has been processed. You can provide information about the lineage when you [create](/cdf/data_governance/guides/datasets/create_data_sets) or [edit](/cdf/data_governance/guides/datasets/edit_explore_data_sets) the data set. # Create data sets and add data Source: https://docs.cognite.com/cdf/data_governance/guides/datasets/create_data_sets Learn how to create data sets in Cognite Data Fusion, configure access control, and add data objects to track data lineage and ensure data integrity. A **data set** is a container for data objects with metadata about the data. For instance, you can use the data set metadata to document who is responsible for the data, upload documentation files, and describe the data lineage. In Cognite Data Fusion (CDF), you'll see data sets as a separate resource type with a `/datasets` API endpoint. To define which **data objects**, such as events, files, and time series, belong to a data set, you specify the relevant `dataSetId` for each data object, typically in a data ingestion pipeline. Data objects can belong to only one data set, so you can unambiguously trace the data lineage for each data object. ## Step 1: Create a data set Navigate to Data fusion > Data catalog. 1. Select **Create data set**. 2. Fill in the basic information and select **Create** to create the data set. Follow the steps in the wizard to fill in **basic information** about the data set, document the [data extraction](/cdf/integration/concepts/extraction) and [transformation](/cdf/integration/concepts/transformation) processes, and add extra **documentation** for your data set. The documentation provides data consumers and data managers with the lineage documentation they need for the data set. We recommend that you upload documentation about how the data is ingested. For example, the documentation could include instructions about how to sign in to a computer where an extractor is running and describe the type of data processing that has been done. You don't have to add all documentation at once. We recommend that you update the information for the data set as you proceed with your data ingestion work. You can also: * Mark the data set as **write protected** to ensure the integrity of the data it contains. * Set **labels** for the data set, for instance, to group similar group data sets and make them more discoverable. * Set the **governance status** for the data set to indicate whether it has a defined owner and follows the data governance processes in your organization. ## Step 2: Set up and scope access to the data set You manage access to the data set **metadata** and the **data objects** within the data set separately: * Use the **datasets** capability to grant users, apps, or services access to add or edit **metadata** for data sets. * Use the **resource type(s)** capabilities to grant users, apps, or services access to add or edit **data objects** within data sets. * Use **scope** to specify which data set(s) the users, apps, or services have access to. * Use **action** to specify what the group is allowed to do with the data they have access to. For example: | Desired access | Capability | Action | Scope | | - | - | - | - | | Read the metadata for **all** data sets. | datasets | read | all | | Read and edit metadata for a data set. | datasets | read

write | dataSetId | | Write-protect a data set. | datasets | owner | dataSetId | | Write time series data to a data set. | timeseries | write | dataSetId | | Read time series data from a data set. | timeseries | read | dataSetId | Users, apps, or services with the capability *datasets* and action *write* can add data sets and edit the metadata. Also, they can edit the metadata even if a data set is write-protected and they're not the owner of the data set. Learn more about managing access for users, apps, and services in the [Access management](/cdf/access) section. We recommend that you: * **Write-protect** data sets that contain production-critical data to preserve the integrity of the data it contains. To write-protect a data set, set the *owner* action for a group and scope it to the `dataSetId`. Only members of groups with the *owner* action can write data to the data set. Other users, apps, or services can't change the data in the data set even if they have the necessary access rights to change the relevant resource types for the data objects contained in the data set. * **Scope write access for 3rd parties** to let them securely write data back to CDF. Create a dedicated data set for the 3rd party and scope its write access to data in that data set. This ensures that they can write data only to that specific data set and not accidentally overwrite any other data. ## Step 3a: Ingest new data into the data set If your extractor writes data to the CDF staging area (RAW) before the data is transformed into the CDF data model, make sure that **the transformation tool** sets the appropriate `dataSetId` of the data objects when it transforms the data. If your extractor writes directly to a **CDF resource type**, make sure that **the extractor** sets the appropriate `dataSetId` when it creates the CDF data objects. Learn more about extractors and transformations in the [data integration](/cdf/integration) section. Get the `dataSetId` for the data set you want to add the data object to. 1. In the left-hand menu, select **Data sets**. 2. Select the data set you want to add the data object to. 3. Copy the **data set ID** from the Basic information section. Copy data set ID from the Basic information section Set the `dataSetId` for the data objects you are ingesting. For example, this request shows how to set the `dataSetId` when you create an *event* in CDF with our API: **URL:** `https://api.cognitedata.com/api/v1/projects/project-name/events` **Body:** ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "externalId": "my.known.id", "startTime": 0, "endTime": 0, "type": "string", "subtype": "string", "description": "string", "metadata": { "property1": "string", "property2": "string" }, "assetIds": [ 1 ], "source": "string", "dataSetId": } ] } ``` ## Step 3b: Add existing CDF data to the data set If you have data objects in CDF that don't belong to a data set, you can update the data objects' `dataSetId` field to include them in a data set, for instance, to trace the data lineage and ensure the data integrity. To update the `dataSetId` field, you can use SQL transformations, a Python script, or another tool of your choice. Get the `dataSetId` for the data set you want to add the data object to. 1. In the left-hand menu, select **Data sets**. 2. Select the data set you want to add the data object to. 3. Copy the **data set ID** from the Basic information section. Copy data set ID from the Basic information section Set the `dataSetId` for the data objects you are adding. For example: To use **the API** to update the `dataSetId` for an *event* with `externalId` 123, use a request like this: **URL:** `https://api.cognitedata.com/api/v1/projects/{project}/events/update` **Body:** ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "update": { "dataSetId": { "set": }, }, "externalId": 123 } ] } ``` To use [SQL transformations](/cdf/integration/guides/transformation/transformations) to set the `dataSetId`, use a SQL statement like this: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT as dataSetId, id FROM _cdf. where ``` For ``, see [this section](/cdf/integration/guides/transformation/transformations#from-other-cdf-resource-types) to find the syntax for the supported CDF resource types. # Edit and explore data sets Source: https://docs.cognite.com/cdf/data_governance/guides/datasets/edit_explore_data_sets Learn how to edit data set information, explore data lineage, and manage data sets in Cognite Data Fusion to ensure data quality and governance. When you or someone else in your organization has [created data sets and added data](/cdf/data_governance/guides/datasets/create_data_sets) to them, you can: * Explore the data sets to trace the **data lineage** to understand where data originates from and be confident of the **data integrity** and the **data quality**. * Set the governance status to **make the data available** as input to the use cases your organization is looking to solve. You can also edit the data sets to add more documentation or to **write-protect** to provide an extra layer of protection to production-critical data. By **archiving** data sets, you can hide data sets that are no longer used. ## Edit data sets Data sets evolve as you ingest more data, update a data ingestion pipeline, or take steps to increase the data set quality. By adding information and documentation about your data set, you provide a shared understanding in your organization about the data in the data set. Navigate to Data fusion > Data catalog. Select the data set you want to edit. Select **Edit**. Use the data set wizard to update the information about the data set. For example, you can: * Edit the **name**, **description**, **owner**, and other data set details to help other users understand the data set content. * Set **labels** for data sets to make the data set easier to discover by others. For instance, you can specify the type of data, geographic region, or business unit in the labels. * Set the **governance status**. If a data set has a defined owner and follows the governance processes for data in your organization, we recommend that you mark the data set as **governed**. If the data set isn't maintained, lacks a clear owner, or lacks necessary governance, mark the data set as **ungoverned**. * Upload additional **documentation files** or specify **links** to further information about the data set. We recommend that you upload documentation about the data ingestion process, such as instructions on how to sign in to a computer where an extractor is running and describe the type of data processing done. * **Write-protect** data sets that contain production-critical data to preserve the integrity of the data. Only members of groups that you explicitly grant access can write data to a write-protected data set. * **Archive** data sets to hide them from the UI and make the data set unavailable to users. The data set and the data it contains aren't deleted, and you can always restore archived data sets later. ## Explore data sets Navigate to Data fusion > Data catalog. To find the data sets you want to explore, you can **filter** data sets by label or **search** for data sets by their name or description. Filter data sets by label in the Data catalog interface 1. Select the data set you want to see more details about. In the **Basic information** section, you can see key information about the data set, for example: * The **data set ID**. This is the CDF ID for the data set. You need to use this ID to add data objects to the data set. * The **name** and **description**. A small padlock in front of the name indicates that the data set is write-protected. * The **Governance status**. "Governed" indicates that the data set has a designated owner and follows the governance processes for data in your organization. * Who **created** the data set. * The **owner** of the data set. * When the data set was last updated. 2. Select the different **tabs** to see more detailed information about the data set: * **Explore data** - see how many data objects of each resource type the data set contains. Select a data object to view more details. * **Lineage** - understand where data originates from and be confident of the **data integrity** and the **data quality**. * **Documentation** - access any uploaded files or documentation links for the data set. * **Access control** - see who has their read or write access scoped to the data set. # About data governance Source: https://docs.cognite.com/cdf/data_governance/index Data governance is a set of principles and practices that ensure high quality through the complete lifecycle of your data and is a must for scalable digitalization in all industries. Cognite Data Fusion (CDF) provides tools and features to make sure your data conforms to the user and organizational expectations: * **Secure access management** to control access for users, apps, and services to the various types of resources (data sets, assets, files, events, time series, etc.) in CDF. **Learn more**: [About access management](/cdf/access) * **Data sets** let you document and track data lineage, ensure data integrity, and allow 3rd parties to write their insights to your CDF project securely. Data sets group and track data by **its source**. For example, a data set can contain all work orders originating from SAP. Typically, an organization will have one data set for each data ingestion pipeline in CDF. Each data object in CDF can belong to only one data set. **Learn more**: [Getting started with data sets](/cdf/data_governance/concepts/datasets) # Data governance overview Source: https://docs.cognite.com/cdf/data_governance/overview_dg Learn about data governance principles, data sets, and how to ensure data quality and integrity throughout the complete lifecycle of your data in Cognite Data Fusion (CDF). ## Get started Learn about data governance principles and practices to ensure high-quality data throughout its lifecycle. Understand how data sets group and track data by source to document lineage and ensure integrity. ## Manage data sets Create data sets, configure access control, and add data objects to track lineage and ensure integrity. Update data set information, explore data lineage, and manage governance status and documentation. # About Data workflows Source: https://docs.cognite.com/cdf/data_workflows/index Orchestrate interdependent processes and tasks in Cognite Data Fusion (CDF) with built-in retry logic, error handling, and failure recovery. **Data workflows** integrates with CDF Transformations, Cognite Functions, API requests, and simulations. This service lets you track execution history, monitor task status, and debug failures. Data workflows can be [triggered on schedules or on data changes](/cdf/data_workflows/triggers). These are the **core concepts of data workflows**: Flowchart showing data workflow core concepts and relationships ## Workflows Workflows are collections of workflow versions, each tied to a **workflow definition** and the workflow version executions. An external ID uniquely identifies a workflow. ## Workflow versions Workflow versions are used for **systematic handling of changes and iterations**. A version is tied to a **workflow definition** and the executions of the workflow versions. Updates to the workflow definition don't automatically require a version change. This means you can decide how to handle versioning and updates. ## Workflow definitions Workflow definitions contain the details about the **tasks to be run and their interdependencies**. The definition outlines the structural layout and progression of the workflow, determining the sequence of tasks to be executed, and serves as a blueprint for the execution process. ## Tasks [**Tasks**](/cdf/data_workflows/task_types#task-types) are **fundamental units of work within a workflow**. Tasks trigger the execution of a process, like running transformations or simulations, calling functions, making API requests, mapping JSON between steps, or orchestrating other tasks. Many parameters support [dynamic references](/cdf/data_workflows/task_types#dynamic-references). For **Transformation**, **Function**, and **Agent** tasks, the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor) helps you build expressions with **custom** path segments in the editor. ## Workflow executions Workflow executions document the **history of runs for a workflow version**. Whenever the `run` endpoint is called, a workflow definition is executed. A workflow execution incorporates particulars such as start time, end time, status, and relevant logs or execution details. It also contains granular task-level execution specifics. This information gives insights into individual tasks' advancement, success, or potential failure and facilitates active monitoring, debugging, and in-depth workflow performance analysis. ## Triggers Triggers automate the execution of workflows based on specific conditions. Instead of manually starting workflows, triggers can run workflows on a schedule or in response to data changes. # Limits and restrictions Source: https://docs.cognite.com/cdf/data_workflows/limits_and_restrictions_workflows Technical specifications and limits for data workflows service in CDF. ## Data workflows service limitations The table row *Task input/output size* limit applies to the input and output of a workflow task. A process triggered by a workflow task, such as a Cognite Function, may have different limits for parameters and inputs. For an overview of limits across CDF services, for example Cognite Functions, see [CDF service limits](/cdf/admin/limits#cdf-service-limits). | Name | Limit per CDF project | | - | - | | Workflows | 1000 | | Workflow definitions | 1000 | | Total number in a single workflow execution, including tasks generated by dynamic tasks | 200 | | Concurrent workflow executions (system-wide) | 50 | | Maximum concurrent runs per workflow (user-configurable) | User-defined | | Concurrent workflow task executions | 50 | | Task input/output size | 0.2 MB | | Triggers | 200 | You can configure the maximum concurrent runs for each individual workflow to control resource usage and ensure predictable processing. This limit operates within the system-wide concurrent execution limit. See [Use data workflows](/cdf/data_workflows/workflow_user_guide#create-data-workflows) for configuration details. ## Workflow execution limitations * Dynamic tasks can't include new dynamic tasks or subworkflow tasks. * Subworkflow tasks can't include dynamic tasks or new subworkflow tasks. * The maximum duration of a workflow execution is 24 hours. * Workflow execution data is stored for three months, unless the total size of the execution payload exceeds 20MB. # Data workflows overview Source: https://docs.cognite.com/cdf/data_workflows/overview Explore guides and references for orchestrating interdependent processes and tasks in Cognite Data Fusion (CDF) with built-in automation, retry logic, and error handling. Use **Data workflows** to orchestrate interdependent processes and tasks in Cognite Data Fusion (CDF), including transformations, functions, and API requests. Understand core concepts including workflows, workflow versions, definitions, tasks, executions, and triggers. Create, run, and monitor data workflows using the CDF user interface, Cognite API, or Python SDK. ## Configure and automate Learn about task types, configure parameters, and use dynamic references to pass data between tasks. Automate workflow execution with schedule-based triggers or data modeling event triggers. ## Reference Review technical specifications and limits for the data workflows service in CDF. # Tasks in data workflows Source: https://docs.cognite.com/cdf/data_workflows/task_types Define and configure tasks that trigger transformations, simulations, functions, API requests, JSON mapping, and more in data workflows. To trigger tasks in a workflow, you must first **define the tasks**. The task definitions are core components of **workflow definitions** and have the following anatomy: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "tasks": [ { "externalId": "", "type": "", "parameters": { ... }, "retries": 3, "timeout": 3600, "dependsOn": [ ... ] } ... ] } ``` | Name | Description | | - | :- | | `externalId` | A unique ID to reference a task within a workflow. | | `type` | The type of task to run: `transformation`, `function`, `simulation`, `cdf`, `jsonMapping`, `dynamic`, or `subworkflow`. | | `parameters` | For each `type`, a distinct set of parameters is required. | | `dependsOn` | The list of tasks to run before this task is triggered. The list is an array of task external IDs. | | `timeout` | The maximum duration a task can run before it's terminated. The default value is 3600 seconds (1 hour). The value is always in seconds. | | `retries` | The number of retries if the task fails. You can set any value from 0 to 10. The default value is 3. | | `onFailure` | Defines how task failures and timeouts will be handled and helps you decide if task completion should be required or optional for the workflow to continue. The default value is `abortWorkflow`, which will mark the workflow as `FAILED` if the task fails or times out. Optionally, set the parameter to `skipTask`, which will mark the task as `COMPLETED_WITH_ERRORS` if the task fails or times out, and continue the workflow execution. | See also the [API specifications](/api-reference/concepts/20230101/workflows) for tasks in data workflows. ## Dynamic references Dynamic references are expressions used to dynamically inject data into **task parameters** during workflow executions. Define references to create [parametric workflows](/cdf/data_workflows/workflow_user_guide#define-workflow-input-data) by passing data from workflow inputs or between tasks. ### Reference picker in the editor For **Transformation**, **Function**, and **Agent** tasks, the **Data workflows** editor provides a **reference picker** to assemble `${...}` expressions without typing them in full. Browse suggested segments and add **custom** segments for nested properties or array indices when values come from runtime data (for example under `${workflow.input}` from [trigger payloads](/cdf/data_workflows/triggers#pass-data-to-workflows)). ### Reference syntax References must follow the format `${prefix.jsonPath}`, where: * **prefix** specifies the data source (workflow input or a task's output or input). * **jsonPath** uses JSONPath-style notation to access nested properties. ### Reference prefixes | Reference prefix | Format | Description | | - | - | - | | Workflow input | `${workflow.input}` | The entire input object passed to the workflow run. Trigger [static and dynamic input](/cdf/data_workflows/triggers#pass-data-to-workflows) is available under this object. | | Task output | `${taskExternalId.output}` | The entire output object from a completed task. Replace `taskExternalId` with that task's **external ID**. | | Task input | `${taskExternalId.input}` | The entire input object passed to that task during the run. | ### Known and custom path segments When you extend a reference beyond its prefix (for example `${workflow.input.items[0].name}`), segments are: * **Known**: Paths the editor can suggest from static configuration or typical output shapes. * **Custom**: Extra segments you supply so the expression matches your JSON at run time (property names, array indices, nested keys), following [JSONPath for nested data](#jsonpath-for-nested-data). ### JSONPath for nested data Use dot notation to access nested properties and bracket notation to access array elements with zero-based indices. This is an example of workflow input data: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "data": { "user": { "email": "user@example.com", "preferences": { "timezone": "UTC" } }, "items": ["item1", "item2"], "matrix": [ ["a", "b", "c"], ["d", "e", "f"] ] } } ``` * `${workflow.input.data.user.email}` → `"user@example.com"` * `${workflow.input.data.user.preferences.timezone}` → `"UTC"` * `${workflow.input.data.items}` → `["item1", "item2"]` * `${workflow.input.data.items[0]}` → `"item1"` * `${workflow.input.data.matrix[0][2]}` → `"c"` * `${workflow.input.data.matrix[1][0]}` → `"d"` ### Make workflows parametric Use references to create reusable workflows that behave differently based on input data: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "process-data", "type": "function", "parameters": { "function": { "externalId": "data-processor", "data": { "source": "${workflow.input.dataSource}", "config": "${workflow.input.processingConfig}", "previous_result": "${prepare-data.output.result}" } } }, "dependsOn": ["prepare-data"] } ``` Not all task parameters support dynamic references. See the [API documentation](/api-reference/concepts/20230101/workflows) for specific parameter support. ## Task statuses During workflow execution, each task progresses through different statuses that indicate its current state: | Status | Description | | - | - | | `SCHEDULED` | The task has been scheduled and is waiting to be picked up by an available worker. | | `IN_PROGRESS` | The task is currently being executed by a worker. | | `COMPLETED` | The task has been successfully completed without any errors. | | `FAILED` | The task failed to complete due to an error. It may be automatically retried if attempts remain within the configured limit. | | `FAILED_WITH_TERMINAL_ERROR` | The task failed to complete due to an error and cannot be automatically retried. | | `TIMED_OUT` | The task did not complete within its configured time limit and was marked as timed out by the system. Note: Any external process initiated by this task might still be running. Check the target system for its status. | | `CANCELED` | The task was canceled without being completed because another task failed, and the workflow was configured to stop on failure. Note: Any external process initiated by this task might still be running. Check the target system for the status. | | `COMPLETED_WITH_ERRORS` | The task encountered errors but is considered complete because it was marked as optional in the workflow version. The workflow continued execution despite the errors in this task. | | `NOT STARTED` | The task is defined in the workflow version but has not yet been reached in the workflow execution. This state is only present in the User Interface of data workflows. | ## Task types ### CDF Transformations tasks Orchestrate CDF Transformation tasks, such as running SQL transformations to process and aggregate data, schedule data ingestions and clean pipelines, or coordinate multiple transformations with dependencies. Set **type** to `transformation`. Under **parameters**, set `externalId` as the external ID of the CDF Transformation to be triggered. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "transformation", "parameters": { "transformation": { "externalId": "" "concurrencyPolicy": "fail", } } } ``` The optional parameter `concurrencyPolicy` determines the behavior of a task if the triggered transformation is already running. The following options are available: | Option | Description | | - | - | | `fail` | The task fails if another instance of the transformation is currently running. This is the default option. | | `waitForCurrent` | The task will pause and wait for the already running transformation to complete. Once the transformation is completed, the task is completed. This mode is useful in preventing redundant transformation runs. | | `restartAfterCurrent` | The task waits for the ongoing transformation to complete. After completion, the task restarts the transformation. This mode ensures that the upcoming/following tasks can use the most recent data. | | `useTransformationCredentials` | This is an optional parameter that determines whether the transformation job will use the credentials specified in the transformation or the credentials of the workflow run. The default value is `false`, meaning the transformation job will use the credentials of the workflow run. | ### Cognite Functions tasks Orchestrate Cognite Functions tasks, such as integrating with external APIs and systems for complex calculations, custom data processing logic, or data validation and quality checks. Set **type** to `function`. Under **parameters**, set the `externalId` as the external ID of the function to be triggered. Use **data** to provide input to the function. Use a JSON string that will be passed as the [data](/cdf/functions/use_functions#call-a-function) argument to the function when it's triggered during the workflow execution. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "function", "parameters": { "function": { "externalId": "", "data": "" } "isAsyncComplete": false, } } ``` **Asynchronous task completion** A Cognite Function task can trigger a process running in a third-party system or prompt a user to take manual action as part of a workflow. Asynchronous completion of Cognite Function tasks uses the `isAsyncComplete` parameter: When `isAsyncComplete` is set to `false` (default), the workflow will trigger the function. The status of the workflow task will reflect the status of the execution of the function. For instance, the task's status will be marked as `COMPLETED` in the workflow when the function has run and completed successfully. When `isAsyncComplete` is set to `true`, the workflow will trigger the function. The status of the workflow task will remain `IN_PROGRESS` until the workflow receives an explicit callback to the update task status endpoint to change the status to one of `COMPLETED`, `FAILED`, or `FAILED_WITH_TERMINAL_ERROR`. You must define the required logic in the Cognite Function to trigger the third-party process and the logic in the third-party process to update the task status in the data workflow. Your Function receives the task ID in its input data under the key `cogniteOrchestrationTaskId`. Use this ID to update the task status in your callback. This is an example of an asynchronous task completion: Diagram showing asynchronous task completion flow with third-party process callback ### Cognite simulation tasks Orchestrate Cognite simulation tasks, such as engineering simulations for process optimization, predictive maintenance simulations to assess equipment health, what-if scenario analysis, or to integrate simulation results into data pipelines. Set **type** to `simulation`. Under **parameters**, set the `routineExternalId` as the external ID of the simulator routine to be triggered. Use the optional `inputs` parameter to override inputs in the routine. It accepts a list of input overrides that will be passed to the routine when it's triggered during workflow execution. Use the optional `runTime` parameter to specify when the simulation should run. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "simulation", "parameters": { "simulation": { "routineExternalId": "", "inputs": "", "runTime": " ### CDF request task Orchestrate a CDF task, such as reading data from CDF APIs to use in subsequent tasks, or to create or update CDF resources dynamically. Set **type** to `cdf`. The task type makes authenticated requests to the Cognite APIs using the credentials provided when the workflow is run. Under **parameters**, fill in the details of the HTTP request to be made: * `resourcePath`: define the endpoint to be called and is prefixed by `{cluster}.cognitedata.com/api/v1/project/{project}` based on the CDF cluster and project. For example, to filter time series, the `resourcePath` would be `/timeseries/list`. * `queryParameters`: enter any additional parameters that should be part of the CDF query. * `method`: give the request `method` - `POST`, `GET`, `PUT` and `DELETE`. * `requestTimeoutInMillis`: is set in milliseconds. There should be a response after triggering a request to the URL. No response from the URL within the timeout interval indicates that the task fails with the status `TIMED_OUT`. * Specifying the `body` of the request is optional. If you use a `GET` method, you don't need the `body` parameter. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "cdf", "parameters": { "cdfRequest": { "resourcePath": "", "queryParameters": "", "method": "", "requestTimeoutInMillis": , "body": "" }, } } ``` ### JSON mapping tasks The features described in this section are currently only available to customers via our **Early Adopter** program. For more information and to sign up, visit the [Early Adopter group](https://hub.cognite.com/groups/json-mapping-task-early-adopter-555) on the [Cognite Hub](https://hub.cognite.com). Orchestrate JSON mapping tasks between steps using [Kuiper](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts) expressions, so you can preprocess or enrich data for downstream tasks such as [**entity matching**](/cdf/integration/guides/contextualization/matching). An external worker evaluates a **Kuiper** expression and returns transformed JSON. For available transforms, see [**Built-in functions**](/cdf/integration/guides/extraction/hosted_extractors/built_in_functions). Set **type** to `jsonMapping`. Under **parameters**, set **jsonMapping** with the following fields. | Field | Required | Description | | - | - | - | | `input` | Yes | JSON object that the mapping receives as input data to transform. | | `expression` | Yes | Kuiper expression that produces the output (transformed JSON) from the declared inputs. Other tasks can read this result through [dynamic references](/cdf/data_workflows/task_types#dynamic-references) using this task’s output. | ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "jsonMapping", "parameters": { "jsonMapping": { "input": { "numericValue": 15 }, "expression": "{\"theAnswer\": input.numericValue + 27}", } } } ``` If the **expression** must contain a literal dollar sign (`$`), escape it as `$$` so it is not parsed as a placeholder. For an overview of how workflow versions carry the task graph and how workflow executions surface per-task inputs and outputs, see [Workflow versions](/api-reference/concepts/20230101-beta/workflow-versions) and [Workflow executions](/api-reference/concepts/20230101-beta/workflow-executions) in the API reference. ### Dynamic tasks Use dynamic tasks when a set of workflow tasks and their execution order are determined at runtime. For instance, dynamic tasks can be used to process variable numbers of data sources, create workflows based on runtime conditions, handle different processing paths based on data content, or create fan-out patterns where the number of parallel tasks depends on input data. The task takes the `tasks` parameter, which must be an array of valid task definitions. The array of tasks must be determined during the runtime of the workflow and can't be included statically in the workflow definition. Instead, use a [reference](#dynamic-references), meaning a dynamic value that will be evaluated when the workflow is run. The dynamic value can either refer to a part of the input to the execution or the output of another task in the workflow. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "dynamic", "parameters": { "dynamic": { "tasks": [ ] } } } ``` This is an example workflow with dynamic task functionality. * The first task of the workflow is a **Cognite Function**, which generates and outputs an array of task definitions. The Python code run in the Function returns a dictionary that includes a `tasks` key. * The second task is a **Dynamic task**, which references and runs the set of tasks defined by the preceding step: `"tasks": "${first-task.output.response.tasks}"`. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { tasks: [ { "externalId": "first-task", "type": "function", "parameters": { "function": { "externalId": "my-cognite-function", } } }, { "externalId": "second-task", "type": "dynamic", "parameters": { "dynamic": { "tasks": "${first-task.output.response.tasks}", } }, "dependsOn": [{"externalId": "first-task"}] } ] } ``` A dynamic task can't start a subworkflow or another dynamic task. ### Subworkflow task Use subworkflow tasks to run a collection of tasks as a workflow within a workflow. This streamlines your workflows by grouping tasks with shared dependencies into a subworkflow or enhances the composability and reusability of your workflows by incorporating subworkflows through references. For instance, you can use subworkflow tasks to group related tasks for better organization, create reusable task blocks within the same workflow, logically separate workflow phases, or isolate error handling for specific task groups. Define these tasks directly in the current workflow definition or reference the tasks to another workflow. #### Inline subworkflow (List of tasks) Use the `tasks` parameter and provide an array of valid task definitions to include a list of tasks directly in the workflow definition: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "subworkflow", "parameters": { "subworkflow": { "tasks": [ ... ] } } } ``` #### External workflow reference Use external workflows to reuse existing workflows across multiple parent workflows, create modular workflow architectures, version and maintain complex workflow components separately, and build workflows from tested, proven components Use the `workflowExternalId` and `version` parameters to reference another workflow and to specify the workflow version to be embedded as a subworkflow: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "subworkflow", "parameters": { "subworkflow": { "workflowExternalId": "my-other-workflow", "version": "v1" } } } ``` Workflows referenced in subworkflow tasks can only nest one level deep. They can't contain subworkflows or dynamic tasks. # Triggers for data workflows Source: https://docs.cognite.com/cdf/data_workflows/triggers Automate workflow execution with schedule-based, data modeling, or records triggers. A data workflow can have one or more triggers associated with it, each uniquely identified by an external ID. See also [API workflow triggers](/api-reference/concepts/20230101/workflow-triggers). ## Pass data to workflows When a trigger starts, it creates a workflow execution and passes data through the workflow input. This data becomes accessible to all tasks in the workflow through [dynamic references](/cdf/data_workflows/task_types#dynamic-references) using the `${workflow.input}` syntax. The workflow input can contain a combination of: * **Static input**: Data you define when creating the trigger. * **Dynamic input**: Data collected by the trigger. For data modeling and records triggers, matching items are available as `${workflow.input.items}`. * **System input**: Metadata added by the system, for instance `${workflow.input.version}`. When you configure **Transformation**, **Function**, or **Agent** tasks, the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor) suggests paths from your trigger's **static input** and other inferable workflow input fields. For deeper runtime shapes (for example inside `${workflow.input.items}`), add **custom** segments as described in [Known and custom path segments](/cdf/data_workflows/task_types#known-and-custom-path-segments). Other task types use `${workflow.input...}` in definitions or the API. ## Set up trigger rules Specify a `triggerRule` to define the conditions for running a trigger. `triggerRule` consists of a trigger type and its associated parameters. ### Schedule a data workflow Use the `schedule` trigger type to run a data workflow at regular intervals. The interval is specified by a [cron expression](https://en.wikipedia.org/wiki/Cron). For example, to run the trigger every day at 12:00 AM, use the cron expression `"0 0 * * *"`. The time zone for the cron expression is UTC. Scheduled triggers pass only the static input data you define when you create a trigger, and system metadata, such as workflow version. ### Trigger on data modeling events Use the `dataModeling` trigger type to run a data workflow based on changes to data modeling instances matching a filter. Specify the filter using the trigger's `dataModelingQuery` parameter. See also [data modeling queries](/cdf/dm/dm_concepts/dm_querying#how-to-define-a-graph-query). Data modeling triggers use a **change-based polling mechanism**: * **Polling**: The system periodically checks for instances matching your filter criteria. * **Change detection**: Triggers detect changes based on the `lastUpdatedTime` of instances. * **Batching**: Multiple matching instances are collected into batches. * **Execution**: When batch criteria are met, a workflow execution starts with the collected instances as input. #### Prevent excessive data modeling trigger runs Data modeling triggers fire on any change that updates the `lastUpdatedTime` property of instances matching your filter. To prevent excessive trigger runs: * **Use specific filters**: Ensure your filter is specific enough to avoid matching instances that shouldn't trigger the workflow. * **Implement processing flags**: Include a property like `isProcessed` in your data model to track workflow completion. * Filter for instances where `isProcessed` is `False`. * Update `isProcessed` to `True` after processing to prevent re-triggering on the same instance #### Control batching You can control the batching of changes using the following parameters: * `batchSize`: The maximum number of items to pass as input to a workflow execution. * `batchTimeout`: The maximum time in seconds to wait for the batch to complete before passing it to a workflow execution. After the `batchTimeout` has passed, a partial batch will be passed to the workflow execution. A complete batch will be passed to the workflow execution without further delay. #### Match `batchSize` with data modeling query limits When using a `dataModelingQuery`, each `with` clause that is referenced in the `select` block **must specify a `limit` equal to the trigger’s `batchSize`**. This ensures that the number of instances returned by the data modeling query matches the size of the workflow input batch. If the `limit` is missing or does not match `batchSize`, the trigger will fail validation. * `with` clauses **referenced** in `select`:\ Must include a `limit` equal to `batchSize`. * `with` clauses **not referenced** in `select`:\ Can omit or use a different `limit`; they are validated separately by the data modeling API. There is **no default `limit`** — it must always be explicitly defined in the query. #### Example: data modeling trigger This example shows how to create a trigger using the Python SDK. The trigger starts a workflow based on changes to instances of `ExampleView` with the property `site` equal to *Site A*. According to the batch configurations, the trigger starts the workflow when it reaches 100 instance changes or 60 seconds after the first change. ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Specify the relevant data model view example_view_id = ViewId(space="ExampleSpace", external_id="ExampleView", version="1") # Define the trigger rule, including the data modeling query trigger_rule = WorkflowDataModelingTriggerRule( data_modeling_query=WorkflowTriggerDataModelingQuery( # Query instances of ExampleView with property site="Site A" with_={ "example_view": NodeResultSetExpression( limit=100, # a with clause referenced by a select should always equal batchSize filter=Equals( property=example_view_id.as_property_ref("site"), value="Site A" ) ) }, # Return properties "site" and "name" from the instances select={ "example_view": Select( sources=[ SourceSelector(source=example_view_id, properties=["site", "name"]) ] ) }, ), batch_size=100, # Start the workflow when reaching 100 changes batch_timeout=60, # Start the workflow when reaching 60 seconds after the first change ) # Create the trigger client.workflows.triggers.upsert( WorkflowTriggerUpsert( external_id="my-data-modeling-trigger", trigger_rule=trigger_rule, workflow_external_id="my-workflow-external-id", workflow_version="v1", ) ) ``` ### Trigger on record changes The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. Use the `recordStream` trigger type to run a data workflow when [records](/cdf/dm/records/concepts/records_and_streams) change in a stream. Specify the stream using the trigger's `streamExternalId` parameter. You can optionally provide a `filter` to limit which records in the stream trigger the workflow, and a `sources` list to select which container properties to include in the workflow input. If you omit `sources`, all properties from the record's containers are included. Records triggers use a **change-based batching mechanism** similar to data modeling triggers: * **Polling**: The system periodically checks for record changes in the specified stream. * **Filtering**: If a filter is defined, only matching records trigger the workflow. * **Batching**: Matching records are collected into batches. * **Execution**: When batch criteria are met, a workflow execution starts with the collected records as input. #### Control batching You can control the batching of changes using the following parameters: * `batchSize`: The maximum number of records to pass as input to a workflow execution. Range: 1-1000. * `batchTimeout`: The maximum time in seconds to wait for the batch to fill before passing it to a workflow execution. Range: 10-86400. After the `batchTimeout` has passed, a partial batch is passed to the workflow execution. A complete batch is passed without further delay. ## Target triggers The trigger targets a data workflow, defined by a `workflowExternalId` and `workflowVersion`. ## Trigger input data You can define an `input` data object as part of the trigger. When the trigger starts executing the target data workflow, this input object will be provided as input to the workflow. ### Static input data Static input data is defined when creating the trigger and remains constant for all executions: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "external_id": "my-trigger", "input": { "environment": "production", "notification_email": "admin@company.com" } } ``` This data is accessible in workflow tasks as `${workflow.input.environment}` and `${workflow.input.notification_email}`. ### Dynamic input: Data modeling triggers For data modeling triggers, the system automatically adds an `items` array to the workflow input containing the instances that matched the trigger criteria: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "version": "v1", "items": [ { "instanceType": "node", "externalId": "instance1", "space": "mySpace", "properties": { "mySpace": { "myView/version": { "site": "Site A", "name": "Equipment 1", "email": "user@company.com" } } } }, { "instanceType": "node", "externalId": "instance2", "space": "mySpace", "properties": { "mySpace": { "myView/version": { "site": "Site A", "name": "Equipment 2", "email": "admin@company.com" } } } } ] } ``` Access this data in your tasks using `${workflow.input.items}`. Each item in the array represents a data modeling instance that matched the trigger criteria, including the properties you selected in the data modeling query. ### Dynamic input: Records triggers For records triggers, the system automatically adds an `items` array to the workflow input containing the records that matched the trigger criteria: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "version": "v1", "items": [ { "externalId": "alarm-001", "space": "mySpace", "properties": { "mySpace": { "alarms": { "severity": "high", "source": "compressor-5", "message": "Temperature threshold exceeded" } } } }, { "externalId": "alarm-002", "space": "mySpace", "properties": { "mySpace": { "alarms": { "severity": "medium", "source": "pump-3", "message": "Vibration anomaly detected" } } } } ] } ``` Access this data in your tasks using `${workflow.input.items}`. Each item in the array represents a record from the stream that matched the trigger criteria. If you specified `sources` in the trigger configuration, only the selected container properties are included. Otherwise, all properties from the record's containers are included. ### Process multiple items Since data modeling and records triggers provide arrays of items, design your workflow to handle multiple items appropriately: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "process-all-items", "type": "function", "parameters": { "function": { "externalId": "batch-processor", "data": { "items": "${workflow.input.items}" } } } } ``` ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "generate-individual-tasks", "type": "function", "parameters": { "function": { "externalId": "task-generator", "data": { "items": "${workflow.input.items}" } } } }, { "externalId": "process-individual-items", "type": "dynamic", "parameters": { "dynamic": { "tasks": "${generate-individual-tasks.output.tasks}" } }, "dependsOn": ["generate-individual-tasks"] } ``` ## Authenticate a trigger The trigger requires a [nonce](/cdf/access/concepts/authentication_flows_oidc#on-behalf-of-flow) for authentication. This is a temporary token that's used for authentication when the data workflow execution starts. A nonce can be retrieved from the [Sessions API](/api-reference/concepts/20230101/function-calls) when creating a session. Make sure that the nonce has the necessary capabilities to create a workflow run and perform the tasks in the workflow. ## Trigger run history You can retrieve a trigger's run history. This gives detailed information about each run, such as when it happened, which data workflow it successfully started, or why the trigger run failed. Use this information to monitor trigger performance and troubleshoot issues with automated workflow executions. The run history includes: * **Fire time**: When the trigger has executed. * **Status**: Whether the trigger successfully started a workflow (`success`) or failed (`failed`). * **Workflow execution ID**: The ID of the workflow execution that was started (if successful). * **Reason for failure**: Details about why the trigger failed. # Use data workflows Source: https://docs.cognite.com/cdf/data_workflows/workflow_user_guide Create, run, and monitor data workflows using Cognite API or Cognite Python SDK. Use Data workflows in the CDF user interface to automate and manage workflow tasks and process runs. ## Before you start To add the necessary capabilities for data workflows, see [assign capabilities](/cdf/access/guides/capabilities#data-workflows). To set up email notifications for workflows, you also need the `subscribesignals:read` and `subscribesignals:write` [capabilities](/cdf/access/guides/capabilities#data-workflows). You can assign a workflow to a [data set](/cdf/data_governance/concepts/datasets) when you create a workflow. All `read` or `write` operations on the workflow or related resources, such as versions, runs, tasks, and triggers, need access to this data set. ## Create data workflows Use Data workflows in the CDF user interface to automate and manage tasks and processes runs. Navigate to Data fusion > Data workflows. Select + Create workflow and follow the wizard to start building your workflow: * Add [workflow tasks](/cdf/data_workflows/task_types#task-types). * Use the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor) when you configure **Transformation**, **Function**, or **Agent** tasks that accept [dynamic references](/cdf/data_workflows/task_types#dynamic-references). * Create [workflow triggers](/cdf/data_workflows/triggers). * Switch between editing the workflow and viewing the workflow run history. * View or modify workflow versions. * Run the workflow, with or without input data. **Control concurrent runs for predictable processing** You can set the maximum number of concurrent runs for each workflow to ensure predictable and consistent data processing. Open **Workflow details** to enter the maximum number of concurrent runs. When the limit is reached, new workflow runs will queue until existing runs complete. ## Run data workflows To run a workflow, call the API endpoint [`/run`](/api-reference/concepts/20230101/workflow-executions), and provide `workflowExternalId` and `version`. ### Authenticate a workflow To start a workflow run, the workflow requires a [nonce](/cdf/access/concepts/authentication_flows_oidc#on-behalf-of-flow). This is a temporary token used for authentication when the workflow triggers the processes defined by its tasks. A nonce can be retrieved from the [Sessions API](/api-reference/concepts/20230101/function-calls) when creating a session. The session will be created behind the scenes when triggering a workflow using the Python SDK. All tasks in the workflow execution use the same nonce for authentication, except for transformation tasks. These tasks are configured with `useTransformationCredentials` set to `true`. The transformation job uses the credentials specified in the transformation. When a trigger creates a workflow run, the nonce configured for the trigger will be passed to the workflow run. Make sure that the nonce has the necessary capabilities to perform the tasks in the workflow. ### Define workflow input data You can define custom `input` data as part of the workflow run. This means workflows can behave differently based on the input ([parametric workflows](/cdf/data_workflows/task_types#make-workflows-parametric)). The input data is accessible throughout the workflow using [dynamic references](/cdf/data_workflows/task_types#dynamic-references). Don't use input data to pass large amounts of data to the workflow. For large data sets, we recommend using CDF's data stores and setting the tasks to directly read from and write to these. See [Limits and restrictions](/cdf/data_workflows/limits_and_restrictions_workflows) for input size limits. The workflow input can come from multiple sources: * **Manual execution input**: Data you provide when manually triggering a workflow. * **Trigger input**: Static data defined when you create triggers. * **Trigger-generated input**: Dynamic data collected by triggers, such as data modeling triggers, add `items` to arrays. * **System input**: Metadata automatically added by the system, such as `version`. ### Add metadata A workflow run can also have custom, application-specific metadata, which is separate from the input data. Use metadata to track execution context, add operational information, and facilitate debugging and monitoring. ## Monitor workflows Track workflow health and receive alerts about execution issues to ensure your workflows run smoothly. ### Workflow execution statuses During execution, each workflow progresses through different statuses that indicate its current state: | Status | Description | | - | - | | `RUNNING` | The workflow is currently running. | | `COMPLETED` | All tasks have completed successfully. | | `FAILED` | The workflow failed to complete due to an error and could not continue to run. | | `TIMED_OUT` | The workflow exceeded its maximum execution time. | | `TERMINATED` | The workflow was manually canceled or stopped by another workflow or process. | ### Email notifications Receive email alerts when workflows fail, time out, or when triggers fail to execute. Users with the appropriate [access capabilities](/cdf/access/guides/capabilities#data-workflows) can configure notifications for any user, while other users can only configure their own settings. To configure notifications, select the workflow external ID in the top bar and choose Notification settings. If you're an admin, select which users should receive notifications. If you're a non-admin user, you can configure your own notification settings. You can also add custom email addresses to receive notifications, for example shared inboxes, or channels in other systems like Slack or Microsoft Teams. Select which events should trigger notifications: * **Workflow execution failed**: Alerts when a workflow run fails * **Workflow execution timed out**: Alerts when a workflow exceeds its user-defined maximum execution time * **Trigger attempt failed**: Alerts when a trigger fails to execute Each user can have different notification preferences. Return to notification settings to update or remove users. ## Task input and output Each task receives a task definition as input during a workflow run. Different outputs are returned depending on the task type: | Task type | Output | | - | - | | **Transformation** | `jobId` of the transformation job in CDF. | | **Function** | `callId`, `functionId`, and `response` of the [Cognite Functions](/cdf/functions). | | **Simulation** | `runId`, `logId`, and `statusMessage` of the [simulation run in CDF](/cdf/integration/guides/simulators/simulator_trigger). | | **CDF** | `statusCode` and `response` (body) of the API call. | | **Dynamic** | `dynamicTasks`, an array of dynamically generated tasks based on the input data. Each task in this array has its own `externalId` and `type`. | These outputs can be referenced by subsequent tasks using [dynamic references](/cdf/data_workflows/task_types#dynamic-references), enabling data flow between tasks. For supported tasks in the user interface, use the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor). For dynamic tasks, refer to the `externalId` of the dynamically generated tasks to access their outputs. **Using client secrets as input to workflows and tasks** Some tasks in a workflow run may need access to client secrets, client credentials, or other confidential information. Data workflows currently don't support secure storage of client secrets as input to workflows or tasks. We recommend leveraging the capabilities for secure storage of secrets in [Cognite Functions](/cdf/functions), combined with a Function task in the workflow. ## Automate workflows with triggers Use triggers to **automate your data workflow runs**. Run workflows at regular intervals for scheduled automation, for instance for daily reports or maintenance tasks or to react to data changes in real-time. Monitor specific conditions and automatically start workflow executions when the conditions are met. Triggers also automatically handle groups of related data changes (batch processing). For details about trigger types, configuration, and how they pass data to workflows, see [Triggers for data workflows](/cdf/data_workflows/triggers) and [API workflow triggers](/api-reference/concepts/20230101/workflow-triggers). ## Error handling and retries Workflows provide built-in error-handling mechanisms: * **Task-level retries**: Each task can be configured with retry attempts. * **Failure policies**: Tasks can be marked as optional (`skipTask`) or required (`abortWorkflow`). * **Compensation flows**: Use subsequent tasks to handle failures or cleanup * **Manual intervention**: Use Function tasks with `isAsyncComplete=true` to create approval points where the workflow waits for human review and manual task completion. For details about task configuration and error handling, see [Tasks in data workflows](/cdf/data_workflows/task_types). # Access control features Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_access_control Learn how to control access to graphs using spaces and access control lists (ACLs) in the Data Modeling Service. Data modeling access control is based on [**spaces**](/cdf/dm/dm_concepts/dm_spaces_instances#space). When you control access to a space, you can control read and write for schemas and instances separately. For each space, you can give some users write access to the schema while allowing others to read the schema and write data instances. ## Access control lists (ACLs) The ACLs used to control access in DMS are: * `dataModelInstances`: controls access to instances (nodes and edges.) * `READ`: allows reading instances. * `WRITE`: allows reading, creating, modifying, and deleting instances. * `WRITE_PROPERTIES`: allows writing properties without allowing creation/deletion of instances. * `dataModels`: controls access to schemas (spaces, containers, views, and data models.) * `READ`: allows reading schemas. * `WRITE`: allows reading, creating, modifying, and deleting schemas. The ACLs support these two scopes: * `all`: grants access to all resources in **all** spaces * `space`: grants access to resources in the specified spaces For example, the capabilities in the example below grant access to: * Read instances in all spaces. * Modify/delete instances in `space1`. * Write properties to instances in `space2` using views/containers you have read access to. * Read and modify schemas in all spaces. ``` - dataModelInstances actions: [READ] scope: all: {} - dataModelInstances actions: [WRITE] scope: space: [space1] - dataModelInstances actions: [WRITE_PROPERTIES] scope: space: [space2] - dataModels: actions: [READ, WRITE] scope: all: {} ``` ## Autocreate instances You can only autocreate instances in spaces you have **write** access to. Even if you set `autoCreateDirectRelations` to `true` when ingesting, the nodes in the spaces you only have read access to **must** already exist. ## Edges and direct relations You need **read** access to the target space to point edges and direct relations to nodes in other spaces. # Aggregation features Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_aggregate Learn how to summarize and analyze data using aggregation functions, grouping, and filters in the Data Modeling Service. Use the [aggregation endpoint](/api-reference/concepts/20230101/instances) to summarize and analyze data by computing advanced metrics and bucketed distributions. For example, you can use the aggregation endpoint to generate data for dashboards and summaries, create numeric summaries, and obtain grouped statistical insights. The aggregation endpoint supports: * **Group by**: partition or group data based on specified properties. For instance, to see the average temperature for equipment, you can group by the `equipment` property. * **Aggregation functions**: compute metrics on the grouped data. This includes calculating sums, averages, minimums, maximums, counts, and histograms. * **Filters and sorting**: like the search endpoint, the aggregation endpoint supports filtering and sorting. This allows you to refine your data set before performing aggregations. ## Recommendations * **Combine filters and aggregations:** to minimize the volume of data to process, always apply filters before running aggregations. * **Use groupBy efficiently:** restrict grouping to properties that are meaningful for your analysis. Excessive grouping can make results difficult to interpret and reduce performance. * **Understand tokenization for text queries:** when searching for text, refer to the search endpoint's [tokenization rules](/cdf/dm/dm_concepts/dm_search#text-analysis-and-tokenization). * **Plan around indexing delays:** because of indexing delays, it may take a few seconds for new or updated data to be available for aggregation. ## Example aggregation request The aggregation request below groups by `equipment` and computes an average of the `temperature` property. ```json title="Example aggregation request" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "view": { "type": "view", "space": "testSpace", "externalId": "Equipment", "version": "v1" }, "query": "temperature sensor", "operator": "AND", "groupBy": ["equipment"], "aggregates": [ { "avg": { "property": "temperature" } } ], "instanceType": "node", "properties": ["name", "description"], "targetUnits": [], "filter": { "range": { "property": "temperature", "lt": 25, "gt": 15 } }, "includeTyping": false, "limit": 100 } ``` Where: * `view`: defines which view the query will read from. Required. * `aggregates`: lists one or more aggregation operations. Maximum five operations in a single request. * `groupBy`: is an array of property names used for grouping. * `query` and `properties`: combine to perform text-based matching on the specified properties (`name`, `description`). * `filter`: restricts data before aggregation. Only results with `temperature` between 15 and 25 are included. ### Aggregating a single type The examples below show different aggregations for a type called **Company** with these properties: | id | name | industry | revenue | head\_count | | - | - | - | - | - | | 2001c18... | Ryan and Friesen and Sons | Internet | 80734000 | 5 | | fe8bf42... | Pfannerstill Inc | Biotechnology | 37038000 | 50 | | 6dc069... | Boyer and Kohler Group | Public Relations and Communications | 21598000 | 1 | | f23a8a... | Mohr Group | Fund-Raising | 7440000 | 56 | | d346e0... | Veum LLC | Real Estate | 5224000 | 10 | #### Average `head_count` across all companies Calculate the average `head_count` across every company in the dataset. ##### Request ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": [ { "avg": { "property": "head_count" } } ], "instanceType": "node", "view": { "type": "view", "space": "Companies", "externalId": "Company", "version": "1" } } ``` ##### Response ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ { "instanceType": "node", "aggregates": [ { "aggregate": "avg", "property": "head_count", "value": 32.29912284331168 } ] } ] ``` #### Average `head_count` grouped by industry Group companies by `industry` and compute the average `head_count` for each distinct industry value. ##### Request ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": [{ "avg": { "property": "head_count" } }], "groupBy": ["industry"], "view": { "type": "view", "space": "Companies", "externalId": "Company", "version": "1" } } ``` ##### Response ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ { "instanceType": "node", "aggregates": [ { "aggregate": "avg", "property": "head_count", "value": 310.941333333333333 } ], "group": { "industry": "Automotive" } }, { "instanceType": "node", "aggregates": [ { "aggregate": "avg", "property": "head_count", "value": 39.10427807486631 } ], "group": { "industry": "Writing and Editing" } } ] ``` #### Histogram aggregation Use the `histogram` function to divide a numeric property (`head_count`) into buckets. Below, each bucket has a width of 10. The result shows how many companies fall into each bucket, grouped by `industry`. ##### Request ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": [ { "histogram": { "property": "head_count", "interval": 10.0 } } ], "groupBy": ["industry"], "view": { "type": "view", "space": "Companies", "externalId": "Company", "version": "1" } } ``` ##### Response (abbreviated) ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "instanceType": "node", "group": { "industry": "Automotive" }, "aggregates": [ { "aggregate": "histogram", "interval": 10.0, "property": "head_count", "buckets": [ { "start": 0.0, "count": 52 }, { "start": 10.0, "count": 65 }, { "start": 20.0, "count": 55 }, ... ] } ] }, { "instanceType": "node", "group": { "industry": "Writing and Editing" }, "aggregates": [ { "aggregate": "histogram", "interval": 10.0, "property": "head_count", "buckets": [ { "start": 0.0, "count": 58 }, { "start": 10.0, "count": 55 }, ... ] } ] } ``` #### Text search within an aggregation Similar to the search endpoint, you can combine text-based queries with aggregations. You can also use the `operator` field to control whether all (`"AND"`) or any (`"OR"`) query terms must match. See [Search features documentation](/cdf/dm/dm_concepts/dm_search) for details. This example computes the average `head_count` for companies where `name` starts with "Inc". ##### Request ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "query": "Inc", "properties": ["name"], "aggregates": [{ "avg": { "property": "head_count" } }], "groupBy": ["industry"], "instanceType": "node", "view": { "type": "view", "space": "Companies", "externalId": "Company", "version": "1" } } ``` ##### Response ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ { "instanceType": "node", "group": { "industry": "Farming" }, "aggregates": [ { "aggregate": "avg", "property": "head_count", "value": 31.4811320754717 } ] }, ... ] ``` ## GraphQL examples You can also perform the examples above through the GraphQL API as well: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Average head_count for all companies: aggregateCompany { items { avg { head_count } } } # Average head_count grouped by industry: aggregateCompany(groupBy: industry) { items { avg { head_count } } } # Histogram of head_count (interval of 10) aggregateCompany { items { histogram(interval: 10) { head_count { count start } } } } # Search by string ("Inc") combined with aggregation: aggregateCompany(query: "Inc") { items { avg { head_count } } } ``` # Containers, views, polymorphism, data models Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_containers_views_datamodels Learn how to define schemas using containers, views, and data models to bring your own properties into the knowledge graph. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). You can create your own schemas to enrich instances with custom properties. The schemas consist of three key elements: * **Containers**: define physical storage which contains properties. * **Views**: establish logical schemas which map properties. * **Data models** are collections of one or more views, used for graph data consumption and ingestion. All three elements are scoped to a space, just like instances: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef SpaceClass fill:#0969DA,stroke:#24292f,stroke-width:2px,color:#fff classDef ViewClass fill:#8250df,stroke:#24292f,stroke-width:2px,color:#fff classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Space([Space]) Container[[Container]] Property([Property]) Index([Index]) Constraint([Constraint]) View[[View]] DataModel([Data model]) Space:::SpaceClass --has-container--> Container Container:::ContainerClass --has-index --> Index Container --has-constraint--> Constraint Space --has-view--> View View:::ViewClass --maps-property--> Property Container --has-property--> Property Space --has-data-model--> DataModel DataModel --contains-view--> View ``` ## Containers Containers are the physical storage for **properties**. They are defined within a space, and hold a set of properties that logically belong together. You must define **types** for your properties, and you can add optional **constraints** that the data must adhere to, and define **indexes** to optimize query performance. You can also define **indexes** for properties or groups of properties to optimize query performance. Containers store properties for instances (nodes and edges.) An instance can have properties in multiple containers: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Equipment[["Equipment
manufacturer (string)"]] Pump[["Pump
maxPressure (float)"]] Node(("Node
externalId: (string)")) Equipment:::ContainerClass --has-data--- Node Pump:::ContainerClass --has-data--- Node ``` You can populate the containers for an instance, in this example below, for a node. This data: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'xyz42' equipment: manufacturer: 'Acme Inc.' pump: maxPressure: 1.2 ``` translates to this: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph LR classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff subgraph types PumpTypeNode(("PumpTypeNode
externalId: pump")) end subgraph equipment Equipment[["Equipment
manufacturer: 'Acme Inc.'"]] Pump[["Pump
maxPressure 1.2"]] Node(("PumpNode
externalId: xyz42
type: [types, pump]")) Equipment:::ContainerClass --has-data--- Node Pump:::ContainerClass --has-data--- Node Node -.-> PumpTypeNode end ``` You can define containers in different space than the space holding the instances. This can be useful if you want to use the same schema for nodes in different spaces, which is often the case given the [access control model](/cdf/dm/dm_concepts/dm_access_control). ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph LR classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff subgraph types PumpTypeNode(("PumpTypeNode
externalId: pump")) end subgraph schema Equipment[["Equipment
manufacturer: (string)"]] Pump[["Pump
maxPressure (float)"]] end subgraph equipment1 Node(("PumpNode
externalId: xyz42
type: [types, pump]
manufacturer: 'Acme Inc.'
maxPressure 2.3")) Equipment:::ContainerClass --has-data--- Node Pump:::ContainerClass --has-data--- Node Node -.-> PumpTypeNode end subgraph equipment2 Node2(("PumpNode
externalId: abc123
type: [types, pump]
manufacturer: 'Other Inc.'
maxPressure 5.6")) Equipment:::ContainerClass --has-data--- Node2 Pump:::ContainerClass --has-data--- Node2 Node2 -.-> PumpTypeNode end ``` As you add data to these containers for more nodes, the physical storage of the containers will look similar to this: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph LR classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Node[[" Node
(space, externalId, type, ...)

(equipment1, xyz42, [types, pump], ...)
(equipment2, abc123, [types, pump], ...)
(types, pump, null, ...)
... "]] Equipment[[" Equipment
(space, externalId, manufacturer)

(equipment1, xyz42, Acme Inc.)
(equipment2, abc123, Acme Inc.)
"]] Pump[[" Pump
(space, externalId, maxPressure)

(equipment1, xyz42, 20.7)
(equipment2, abc123, 14.2)
"]] Equipment:::ContainerClass --> Node:::ContainerClass Pump:::ContainerClass --> Node Pump -.-> Equipment ``` Note that only `node.{space, externalId, type}` is included in the `Node` base container for brevity. This is similar to relational database schemas where `(space, externalId)` constitutes a foreign key to the core node table, and results in a [snowflake schema](https://en.wikipedia.org/wiki/Snowflake_schema). Importantly, this data lives on a different plane than the graph data discussed in the previous section. For example, nothing ensures that a node has data in `Pump` just because it has `node.type` set to `[types, pump]`. Validation of data content is left to the client to determine, but you can use [**views**](#views) to make it more ergonomic. ### Which types of instances can you use a container for? The `usedFor` field lets you define which types of instances the containers can be used for. Specify one of these values: * `node`: the container can only be used to populate properties on nodes. * `edge`: the container can only be used to populate properties on edges. * `all`: the container can be used to populate properties on both nodes and edges. * `record`: the container can only be used for [records](/cdf/dm/records/concepts/records_and_streams). You can only ingest records into containers with `usedFor` set to `record`. Containers designated for records support significantly more properties than standard containers. See [Limits and restrictions](/cdf/dm/dm_reference/dm_limits_and_restrictions). If you use `all`, ingesting to the container will be more expensive than using only `node` or `edge`. ### Properties When you define a container, you must specify the properties it will contain. Data modeling supports the following basic data types for properties: | Property type | Description | | - | - | | `text` | A string of characters. | | `int64` | A 64-bit integer. | | `float64` | A 64-bit floating point number. | | `float32` | A 32-bit floating point number. | | `boolean` | A boolean value. | | `timestamp` | A timestamp (with timezone). | | `date` | A date (without timezone). | | `json` | A JSON object. | | `direct` | A direct relation to another instance. | | `enum` | A fixed set of named values. | In addition to these property types, we support native reference types that point to resources in other Cognite APIs. This lets you reference data not suited for storage in a property graph. We support the following native resource reference types: | Native resource reference type | Description | | - | - | | `TimeSeries` | A reference to one specific time series. You can use GraphQL queries to expand data from the time series, including data points. | | `File` | A reference to a file stored in CDF and uploaded through the files service. | | `Sequences` | A reference to a sequence stored in CDF. | We support declaring all of these base and reference types as **lists**. For example, to store a list of file references: `files: [File]` You can specify whether the property is **nullable**, **immutable**, and provide a **default value**. Marking a property as immutable means that its value cannot be changed after it is set, although it is possible to toggle this setting. The full specification of a required string property can look like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: myStringProperty description: A string property nullable: false immutable: true defaultValue: foo type: type: text list: false ``` When creating your containers, it is important to consider how they will be queried. Identify the properties likely to be used to filter and sort on when querying, and create indexes or composite indexes to support your queries. Since indexes can represent additional update overhead when data is mutated or ingested, make sure the indexes you create are useful and necessary with testing and performance validation. It is possible to set `nullable` to false on an existing property. New null values will be rejected, but existing null values remain. To check for existing null values, fetch the container and check the state fields of the property (see [Background validation](#background-validation)). ### Indexes Well-designed indexes speed up data access, they support constraints such as uniqueness, and efficient cursoring with custom sort operations. Data modeling supports up to 10 indexes per container. An index belongs to the container and is not a flag on a property. When you're laying out your physical schema, it's important to remember that you can only build indexes using properties from the same container. Indexes cannot be built from properties hosted in different containers. We support two index types: * `btree`: Use a btree index on primitive base types for efficient lookups and range scans. You can set btree indexes to be `cursorable`, and enable efficient cursoring with custom sorts. Set btree indexes to `bySpace`, indicating that they'll include the `node.space` property as a prefix on the index. * `inverted`: Use an inverted index for list-type properties to enable efficient searching for values that appear within the list. After an index is created through an API call, the service will start building the index in the background. To check the progress, fetch the container and check the `state` field of the index (see [Background validation](#background-validation)). #### Size bounds on btree indexes B-tree indexes are less effective for very large property values. Large indexed values can exceed datastore limits and cause data ingestion to fail. Large pre-existing values can cause new indexes to fail to build. To prevent ingestion failures and guard against broken B-tree indexes, you should limit the size of indexed properties. You can bound the property size in two ways: * `maxListSize` limits the number of elements in a list property. The limit can be set as high as 2000. To be included in a B-tree index, the limit must be at most 300 or 600, depending on the property type. * `maxTextSize` limits the number of UTF-8-encoded bytes in a text property. The limit can be set as high as 128k. To be included in a B-tree index, the limit mustn't exceed 2400 bytes. If a btree index has several indexed properties, the combined limits on all the properties must not exceed 2400 bytes. You must set `maxListSize` to include a property in a new B-tree index. We currently allow properties without `maxTextSize` to be included in B-tree indexes, but this will be disallowed in the future. Clean up existing data that exceeds size bounds before creating B-tree indexes. Oversized data can break new indexes and stop them from improving query performance. ### Constraints Use constraints to restrict the values that can be stored by a property, or in a container. Constraints ensure that the data has integrity, and reflects the real world. We support up to 10 constraints per container. Currently, we support two constraint types: * `uniqueness`: Ensures that the values of a property or a set of properties are unique within the container. Set a uniqueness constraint to `bySpace`, indicating that the uniqueness will apply per space. * `requires`: Points to another container, and requires that the instance has data in that other container used to populate this container. ### Background validation Creating a new constraint or index is an expensive operation that requires scanning through all existing data. Therefore, the creation endpoints return immediately without waiting for the new constraint or index to be fully valid. You won't know whether the constraint or index is fully built and valid without checking back later. To track the progress of this background work, constraints, indexes and properties returned from the service have read-only state fields, which describe the progress of the background work. State fields have three possible values: * `"current"`: The constraint is validated, or the index is built. No problems were detected. The index is ready for use. * `"failed"`: Data exists that violates the constraint, or prevents the index from being built. A failed index can't be used to speed up queries. * `"pending"`: The background work is still ongoing. Check back later. Constraints are still enforced on new data even if they are in the `failed` state. Some constraints are implicitly defined as fields on the property object, instead of being constraint objects in their own right. To track the state of these constraints, properties returned from the service have a `constraintState` field, which contains three state fields: * `nullability`: Tracks the state of the nullability constraint when setting `nullable` to `false`. * `maxListSize`: Tracks the state of the size bound constraint on list properties when setting `maxListSize` to a non-null value. * `maxTextSize`: Tracks the state of the size bound constraint on text properties when setting `maxTextSize` to a non-null value. These state fields are `null` if the property does not have the corresponding constraint. The service does not automatically revalidate or rebuild a constraint or index after it reaches the `failed` state. After you resolve the problem that caused the failure, for example by removing the pre-existing data that violated a given constraint, you can manually request a revalidation by using one of three endpoints: * [`indexes/retry`](/api-reference/concepts/20230101/containers): Retry indexes that failed to build. * [`constraints/retry`](/api-reference/concepts/20230101/containers): Revalidate requires or uniqueness constraints that had invalid pre-existing data. * [`properties/retry`](/api-reference/concepts/20230101/containers): Revalidate nullability or size-bound constraints on properties that had invalid pre-existing data. These endpoints trigger a new scan through all relevant data, so use them sparingly. ### Example container definition This example defines two containers, `Equipment` and `Pump`: * It sets `usedFor` to `node` on both containers to allow them to be populated for nodes, not edges. * The `btree` index on `Equipment.manufacturer` enables efficient sorting/filtering on the `manufacturer` property. The index is cursorable and lets you efficiently cursor through equipment nodes when sorting on `manufacturer`. * The `requires` constraint on `Pump` ensures that any node with data in the `Pump` container also has data in the `Equipment` container. ```yaml title="Pump-Equipment definition" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: equipment externalId: Equipment usedFor: node properties: - manufacturer: type: type: text list: false nullable: false indexes: manufacturer: type: btree properties: - manufacturer cursorable: True - space: equipment externalId: Pump usedFor: node properties: - maxPressure: type: type: float64 list: false nullable: false constraints: requireEquipment: constraintType: requires require: space: equipment externalId: Equipment ``` ## Views Use **views** to create logical **schemas** to consume and populate a graph tailored for specific use cases. Like containers, views contain a group of properties. You define the views by either mapping container properties, or by creating connection properties to express the expected relationships in the graph. You query data through your defined views. Data is not queried directly from the containers. ### Mapped properties Views let you **map properties** from different containers in a "flat" object and rename or alias properties. For example, this view creates a flat object with the properties `manufacturer` and `maxPressure` from the `Equipment` and `Pump` containers. It also renames the `manufacturer` property to `producer`: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef ViewClass fill:#8250df,stroke:#24292f,stroke-width:2px,color:#fff classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Equipment[["Equipment
manufacturer (string)"]] PropertyManufacturer([manufacturer]) Pump[["Pump
maxPressure (float)"]] PropertyPressure([maxPressure]) BasicPump[["BasicPump
producer (string)
maxPressure (float)"]] Equipment:::ContainerClass --has-property--> PropertyManufacturer Pump:::ContainerClass --has-property--> PropertyPressure BasicPump:::ViewClass --maps--> PropertyManufacturer BasicPump --maps--> PropertyPressure ``` You can use the view to populate a node with data from both the `Equipment` and `Pump` containers at the same time, and query for the properties when retrieving the nodes. ### Connection properties Connection properties let you describe that you expect certain direct relations or edges to exist between nodes in the graph. When this metadata is persisted, you can retrieve related data when consuming instances through a particular view. #### Reverse direct relation Defining a reverse direct relation property between two data types is useful for describing the connections that exist in the graph. In the example below the Equipment has a direct relation to a Manufacturer. Since the relation property is part of the Equipment properties, the manufacturer view can describe the reverse direct relation. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef ViewClass fill:#8250df,stroke:#24292f,stroke-width:2px,color:#fff classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Manufacturer[["Manufacturer
name (string)"]] ManufacturerView[["Manufacturer
name
equipment"]] Equipment[["Equipment
name
manufacturer (Manufacturer)"]] EquipmentView[["Equipment
name
manufacturer"]] ManufacturerView:::ViewClass --maps--> Manufacturer:::ContainerClass EquipmentView:::ViewClass --maps--> Equipment:::ContainerClass ManufacturerView --reverse direct relation--> EquipmentView EquipmentView --direct relation--> ManufacturerView ``` * `connectionType`: Either `single_reverse_direct_relation` or `multi_reverse_direct_relation`, depending on if you expect a single relation or multiple objects. * `name`: The name of the property. * `description`: The description of the property. * `through`: The source and identifier of the direct relation property. * `source`: The source definition of the expected data. * `targetsList` (read-only): Whether the reverse direct relation targets a *list* of direct relations or not. The `single_reverse_direct_relation` doesn't ensure that **only** a single instance is connected. If you want to ensure only a single instance can be connected you need to ensure that the Direct Relation property has a uniqueness constraint. As reverse direct relations traverse the graph, it is highly recommended that there is a b-tree index on the direct relation property. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} equipments: name: 'Equipments' description: 'All equipments made by the manufacturer' connectionType: multi_reverse_direct_relation through: identifier: manufacturer source: type: view space: vendor_schema externalId: Equipment version: 1 source: type: view space: vendor_schema externalId: Equipment version: 1 ``` #### Edges You can also define a connection using an edge. For example, you can express that you expect nodes with data in `BasicPump` to have `flows-to` edges to `Valve` nodes. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef ViewClass fill:#8250df,stroke:#24292f,stroke-width:2px,color:#fff classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff Equipment[["Equipment
manufacturer (string)"]] PropertyValves([valves]) Pump[["Pump
maxPressure (float)"]] Valve[["Valve
state (string)"]] BasicPump[["BasicPump
producer (string)
maxPressure (float)
valves: [BasicValve]"]] BasicValve[["BasicValve
producer (string)
state (string)"]] BasicPump:::ViewClass --has-connection--> PropertyValves BasicPump --maps--> Pump:::ContainerClass BasicPump --maps--> Equipment:::ContainerClass BasicValve:::ViewClass --maps--> Valve:::ContainerClass BasicValve --maps--> Equipment PropertyValves -.-> BasicValve PropertyValves -.-> flowsTo subgraph types flowsTo((flows-to)) end ``` The example above encodes that nodes with data in `BasicPump` can have `flows-to` edges to nodes with data in `BasicValve`. You can describe this with these fields in a connection property: * `connectionType`: Supports `single_edge_connection` or `multi_edge_connection` (default). * `type`: the fully qualified external ID of the node representing the edge type. * `source`: a reference to the view which you can view the node in the other end through. * `direction`: the direction to traverse the edge (inwards/outwards). * `edgeSource`: an optional reference to a view. A Pump/Valve would look like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: space: types externalId: flows-to source: space: equipment externalId: BasicValve version: v1 direction: outwards ``` ### Implementing other views Views can implement other views and inherit their properties. This is useful when you want to create a view that combines the properties of multiple other views. For example: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: equipment externalId: Equipment version: v1 properties: manufacturer: container: space: equipment externalId: Equipment containerPropertyIdentifier: manufacturer - space: equipment externalId: Pump version: v1 implements: # <-- Declares that the view implements the Equipment view - space: equipment externalId: Equipment version: v1 properties: maxPressure: container: space: equipment externalId: Pump containerPropertyIdentifier: maxPressure ``` The effective properties of the `Pump` view are now `manufacturer` *and* `maxPressure`. The effective properties of a view are resolved at query time. We do not allow breaking changes to views, but if a view implemented by another view is deleted, the inherited properties will be removed from the implementing view. This could break clients if it removes any required properties. #### Implemented property conflicts and precedence If you, for example, have four views; A, B, C, and D, each with a single property with the following implements graph, you can see the effective properties on the right. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph TD subgraph effective properties EA("A(a,b,c,d)") --> EB("B(b,c,d)") EB --> EC("C(c)") EB --> ED("D(d)") end subgraph properties A("A(a)") --> B("B(b)") B --> C("C(c)") B --> D("D(d)") end ``` If you introduce conflicting property identifiers in this graph, they are resolved by sorting the `implements` graph topologically. The order beneath a node is determined by the order of the implements array, where later entries are preferred. If `B implements [C, D]`, the order of precedence is `A, B, D, C`. If `B implements [D, C]`, the order is `A, B, C, D`. In these examples `B implements [C, D]`: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph TD subgraph effective properties EA("A(a,b,c,d,D.x)") --> EB("B(b,c,d,D.x)") EB --> EC("C(c,C.x)") EB --> ED("D(d,D.x)") end subgraph properties A("A(a)") --> B("B(b)") B --> C("C(c,x)") B --> D("D(d,x)") end ``` ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph TD subgraph effective properties EA("A(a,b,c,d,A.x)") --> EB("B(b,c,d,C.x)") EB --> EC("C(c,C.x)") EB --> ED("D(d)") end subgraph properties A("A(a, x)") --> B("B(b)") B --> C("C(c,x)") B --> D("D(d)") end ``` ### View versioning Views are versioned, and you can not introduce breaking changes without changing the version. See [view versioning](#view-versioning) and [data model changes](#data-model-changes) sections for information about data model and view changes that force a version update. You can adapt the versioning scheme to your needs. If you don't have a preference, we recommend using an integer (whole numbers). Start your version scheme at 1, and increment the version number by one for each new version. You can use decimals — 1.1, 1.2, and 2.0 — if you want higher version granularity. In this example, the **first digit** typically increments each time you make a **significant** or a **breaking** change. Breaking changes could require changes to your application's business logic. The **fractional number** — 2nd and 3rd digits — increments when you make **minor** changes that doesn't break your API. These kinds of changes don't typically break the logic required to use the API (and models). [Semantic versioning](https://semver.org) is a widely used versioning scheme. All version increments to a data model will break, as non-breaking changes are allowed without changing the version. You can choose to implement semantic versioning, but doing so requires close attention to the reusability factor designed for CDF data modeling. The semantic versioning scheme is not designed for data modeling but for software development. The `a.b.c` versioning scheme in semantic versioning specifies that both changes in `a` and `b` are breaking changes. This leaves only `c` for non-breaking changes. In CDF, we allow certain changes to the data model without forcing you to increment the version number for the view or data model. For example, adding a new data type to an existing data model isn't a breaking change. However, in semantic versioning, this would be a breaking change. ### View filters All views have a filter field that lets you filter the nodes that are included when querying the view. For most higher-level query endpoints in DMS, the filters are applied automatically. For advanced endpoints, you have to apply the filters manually. If no filter is specified, the default `hasData` filter is applied on the list of views specified when querying. Learn more about `hasData` filters in the [querying](/cdf/dm/dm_concepts/dm_querying) article. ### Equipment example view This example illustrates a view definition for equipment: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: equipment externalId: BasicEquipment properties: producer: container: space: equipment externalId: Equipment containerPropertyIdentifier: manufacturer - space: equipment externalId: BasicValve version: v1 # Since this only maps properties in the Equipment view, we can't rely on hasData filtering. # We add a custom filter to make sure we only include nodes of the correct type. filter: equals: property: ['node', 'type'] value: { 'space': 'types', 'externalId': 'valve' } implements: # Inherit the properties from the BasicEquipment view - space: equipment externalId: BasicEquipment - space: equipment externalId: BasicPump version: v1 implements: # Inherit the properties from the BasicEquipment view - space: equipment externalId: BasicEquipment properties: maxPressure: container: space: equipment externalId: Pump containerPropertyIdentifier: maxPressure valves: type: # The edge type to traverse space: types externalId: flows-to source: # The view to view the other node in space: equipment externalId: BasicValve version: v1 direction: outwards ``` ### Caveats * When a property mapped by a view is modified or deleted, the `lastUpdatedTime` of the view will *not* be updated. * It's important to inspect which views are affected by the deletion of container. When a container is deleted, its properties are removed from any views that map the container. This can break clients if the properties are required. ## Polymorphism in views You can achieve polymorphism for views in two ways: * [Using `implements` and the implicit view `hasData` filter](#using-implements-and-implicit-view-hasdata-filtering). * [Using explicit view filters on `type`](#using-explicit-view-filters-on-type). In the sections below, `BasicPump` and `BasicValve` are subtypes of `BasicEquipment`. ### Using `implements` and implicit view `hasData` filtering When the `BasicPump` view implements the `BasicEquipment` view, the default filter on `BasicPump` is a `hasData` filter across the underlying containers: `Pump` and `Equipment`. If you filter using the `BasicEquipment` view, you'll get anything with data in the `Equipment` container. If you filter using the `BasicPump` view, you'll get anything with data in both the `Equipment` *and* the `Pump` containers. This approach to polymorphism resembles **structural** subtyping: "*if it looks like a duck and quacks like a duck, it's a duck.*" This breaks down in some cases. For example, if a view only has connection properties, there are no backing containers to apply the `hasData` filtering on. In this case, you can [use the `type` property and view filters](#using-explicit-view-filters-on-type) - see the next section. ### Using explicit view filters on `type` If, for example, you associate all pump nodes with the type `[types, pump]`, you can use a view filter to only include nodes with that type: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: equals: property: ['node', 'type'] value: { 'space': 'types', 'externalId': 'pump' } ``` To list all equipment nodes, you must explicitly include the subtypes of the view in your filter: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: in: property: ['node', 'type'] values: [ { 'space': 'types', 'externalId': 'pump' }, { 'space': 'types', 'externalId': 'valve' }, ] ``` This approach to polymorphism resembles **nominal** subtyping: "*if it's a duck, it's a duck.*" ## Data models Use **data models** to **group views** that belong together for a purpose. For example, you might define a `EquipmentInspection` data model containing a `BasicValve` and a `BasicPump` view. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} space: equipment externalId: EquipmentInspection version: v1 views: - space: equipment externalId: BasicPump version: v1 - space: equipment externalId: BasicValve version: v1 ``` ## Impact of changes to views and data models The tables in this section describe the impact of changes to a container, a view, or a data model. Changes that force a visible change for a consumer of data from a view or a data model are considered **breaking changes**, and require a version change to be submitted to the service. You can change the version for a data model, or view even if a version change is *not* required. This operation should be considered a controlled switch for any consumer of data from the configuration. ### Container changes Containers are not versioned like views and data models, but you can still evolve them. The possible operations on a container fall into one of three categories: * Non-breaking and allowed * Breaking but allowed * Breaking and disallowed To perform a disallowed operation on a container, use an expand-and-contract pattern: 1. Create a new container with the desired spec. 2. Re-ingest the property data into the new container for the relevant instances. 3. Create new view versions that map to these containers, and, optionally, new data model versions to include the new views if necessary. 4. When the old views (still mapping the old containers) are no longer in use, delete the old containers and views. After deleting, recreating, re-ingesting or creating a parallel container, you need to recreate any existing view-to-container mapping(s). In the table below, the "*Not allowed*" description means that the operation can't be safely implemented in the underlying data store. | Operation | Breaking change | Allowed | Description | | - | - | - | - | | Change name. | No | Yes | Changes metadata. | | Change description. | No | Yes | Changes metadata. | | Change `usedFor`. | N/A | No | Not allowed. | | Add property. | No | Yes | When the property identifier isn't in use. | | Delete property. | N/A | No | Not allowed. | | Add `requires` or `check` constraint. | No | Yes | Identifier must be unique and constraint will apply to any new values (not pre-existing ones). | | Add `unique` constraint. | N/A | Yes | Can only be included during the initial creation of the container. | | Change constraint. | N/A | No | Not allowed. | | Delete constraint. | No | Yes | Allowed. | | Add new index. | No | Yes | Allowed. | | Delete index. | No | Yes | Allowed. | | Change index. | N/A | No | Not allowed. | | Change property: `nullable` > `non-nullable`. | Yes | Yes | Changing to non-nullable may break ingestion clients, but represents an acceptable breakage. This shouldn't be unexpected. | | Change property: `non-nullable` > `nullable`. | N/A | No | Not allowed. | | Change property: `autoIncrement`. | N/A | No | Not allowed. | | Change property: `defaultValue`. | No | Yes | Changes are applied to new values added. Previously existing values remain the same. | | Change property: `description`. | No | Yes | Changes metadata meant for human consumption. | | Change property: `name`. | No | Yes | Changes metadata meant for human consumption. | | Change property: `type`. | N/A | No | Not allowed. | | Change (text) property: `list` state. | N/A | No | Not allowed. | | Change (text) property: `collation`. | N/A | No | Not allowed. | | Change (primitive) property: `list` state. | N/A | No | Not allowed. | | Change (direct relation) property: target `container`. | N/A | No | Not allowed. | | No change. | No | Yes | Allows idempotent updates. | ### View changes Operations and the impact of the operation for data modeling views. | Operation | View | Property | Relation | Direct relation | Breaking change | Description | | - | - | - | - | - | - | - | | Change name. | Yes | Yes | Yes | Yes | No | Changes metadata. | | Change description. | Yes | Yes | Yes | Yes | No | Changes metadata. | | Change filter. | Yes | Yes | Yes | Yes | No | Changes the result of the view, but doesn't change the form of it. | | Change implements. | N/A | N/A | N/A | N/A | Yes | May break clients. | | Change version. | N/A | N/A | N/A | N/A | Yes | A version can be bumped even if there are no changes (allows alignment of version numbers for different combinations of views). | | Add nullable property. | Yes | Yes | N/A | N/A | No | Safe change as long as there's no collision with other inherited properties, and it doesn't introduce a new mapped container. Still safe as long as the type is the same and the property doesn't change from `nullable` to `required`. | | Add non-nullable. | Yes | Yes | N/A | N/A | Yes | Breaking since a client may depend on its existence. | | Delete property. | Yes | N/A | N/A | N/A | Yes | | | Change property type. | N/A | Yes | N/A | N/A | Yes | Version change to not break consumers. | | Change container reference for base property. | N/A | Yes | N/A | N/A | No | Permit remapping consumers to new data, from other containers, without forcing a version change for consumers. | | Change source hint. | N/A | N/A | Yes | Yes | Yes | Equivalent of changing the property type. | | Change type of relation. | N/A | N/A | Yes | Yes | Yes | As long as the source type remains the same (not changed). | | Change direction of relation. | N/A | N/A | Yes | Yes | Yes | As long as the source type remains the same (not changed). | | Change source of relation. | N/A | N/A | Yes | N/A | Yes | As long as the source type remains the same (not changed). | | No change. | N/A | N/A | N/A | N/A | No | Support idempotent updates. | ### Data model changes Operations and the impact of the operation for data models in the data modeling service. | Operation | Breaking change | Allowed | Description | | - | - | - | - | | Change name. | No | Yes | Changes metadata. | | Change description. | No | Yes | Changes metadata. | | Add a view. | No | Yes | Non-breaking if the new view doesn't conflict with any existing views in the data model. A conflict is a view using the same external ID. | | Remove a view. | Yes | Yes | Breaking since a client may depend on the existence of the data model. | | Replace a view. | Yes | Yes | When updating the view identified by the (`space`, `externalId`, `version`). Typically done to update the version of a view used by the data model. | | Change version. | Yes | Yes | A version bump without other changes is still a version bump. | | Change space. | N/A | No | Not supported. | # Ingestion features Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_ingestion Learn how to ingest and remove instances in your graph using apply, patch, replace, and delete operations. This section describes how to ingest and remove instances in a graph using the [/model/instances](/api-reference/concepts/20230101/instances) and [/delete](/api-reference/concepts/20230101/instances) endpoints. ## Applying instances This example `apply` request attempts to ingest a node with the external ID `pump42` into the space `equipment`, and set the `Pump/v1.producer` view property to "Acme Inc.". ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: - instanceType: node space: equipment externalId: pump42 sources: - source: type: view space: equipment externalId: Pump version: v1 properties: producer: 'Acme Inc.' ``` You can also ingest properties directly into containers. All instances in a request are applied atomically, and you can apply a mix of nodes and edges. Consequently, all instances in a request are applied, or the entire request fails. ### Create, patch, and replace When you apply instances, you can use three distinct modes depending on the existing state and the request parameters. * If the instance does not exist at the time of ingestion, it will be **created**. This means that all required properties must be set. * If the instance already exists, you have two updating methods: **`patch`** and **`replace`**. You choose the method by setting the value of the `replace` flag. The value can be `true` or `false`, and the default value is `false`. With `patch`, only the properties that are provided in the `sources` section of the request will be updated, leaving all other properties unchanged. With `replace`, all properties will be updated, and omitted properties will be set to null. Consequently, all required properties must have a value specified in the request body when you update an instance with `replace=true`. ### Autocreate nodes When ingesting instances, you can autocreate the targets of direct relation properties and ingest a graph without worrying about the order in which you ingest the instances. Use these properties to control this behavior: * `autoCreateDirectRelations`: automatically create all missing direct relation target nodes (default: `true`.) * `autoCreateStartNodes`: when set to `true`, automatically create any missing start nodes for the edge (default: `false`.) When the parameter is set to `false` and the indicated start node is not present, the request will fail with a HTTP status code of 409, indicating a conflict with the current state of the service. * `autoCreateEndNodes`: when set to `true`, automatically create any missing end nodes for the edge (default: `false`.) When the parameter is set to `false` and the indicated end node is not present, the request will fail with a HTTP status code of 409, indicating a conflict with the current state of the service. You need write access to a space to autocreate nodes in it. ### Optimistic concurrency control Every instance has a version number. It starts at `version: 1` and increments by 1 each time the instance is changed. The `apply` endpoint supports optimistic concurrency control and allows you to specify an expected **existing** version on the instance. If an actual version is greater than the expected version, the request fails with status code `409`. The following example creates a pump node with the external ID "pump42". We set `existingVersion` to `0` to make sure the node does not exist, and that you will be **creating** the node. If the node `equipment.pump42` already exists, the request will fail. If it doesn't exist, the node is created. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: - instanceType: node space: equipment externalId: pump42 existingVersion: 0 ``` The next example illustrates a case where you want to update the node `equipment.pump42` and set the `Pump/v1.producer` property to "Acme Inc.", but only if the `producer` property is currently null, to prevent you from overwriting someone else's changes. First, read the node and verify that the `producer` property is null and then apply the update specifying `existingVersion=$currentVersion`. If the first read request returns `version: 1`, you can apply the instance: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: - instanceType: node space: equipment externalId: pump42 existingVersion: 1 sources: - source: type: view space: equipment externalId: Pump version: v1 properties: producer: 'Acme Inc.' ``` If someone has set the `producer` property in the meantime, their request has bumped the node's version to 2. Because you specified `existingVersion: 1`, but the actual value is `version: 2`, the request fails with status code `409`. You can read the node again, check if the `producer` property is null, and repeat the process with `existingVersion` set to 2. You can also **not** fail the entire request when there are version conflicts, and instead ignore updates to the instances that have version conflicts. You do this by setting `skipOnVersionConflict` to `true` in the request. Sending an update request that will **not** update the instance's properties will **not** bump the `version` or `lastUpdatedTime` of the instance. ## Deleting instances If you have stored data across multiple containers using the **same** space and externalId combination, calling delete will delete all the data pertaining to the node. To delete nodes and edges from the graph, you send a list of fully qualified external IDs to the `/delete` endpoint. This example deletes a node and an edge: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: - instanceType: node space: equipment externalId: pump42 - instanceType: edge space: equipment externalId: i-dont-exist ``` The response from this request contains only the successfully deleted instances. In this case, only the node `equipment.pump42`, since the `equipment.i-dont-exist` edge does not exist: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: - instanceType: node space: equipment externalId: pump42 ``` ### Soft deletion To support syncing deletes to secondary stores, instances are always soft deleted at first. The soft-deleted instances will be returned by the [/sync](/cdf/dm/dm_concepts/dm_querying#syncing---subscribing-to-changes) endpoint with `deletedTime` set to a non-null value. After a grace period (3 days by default), the instances are deleted by the system. Soft deleted instances count towards the total instance [limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#schema-and-instance-limits) for you CDF project. You can not restore soft deleted instances. ### Delete edges before nodes Deleting a node will delete all edges connected to that node. It is a best practice to delete the edges before the nodes in order to avoid long running cascading deletes. It is critical to delete edges before the nodes when you have nodes with a large number of edges pointing to/from them. # Schema and instance inspection Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_inspect Learn how to inspect schemas and instances to understand their structure and relationships in your data model. Inspection endpoints offer detailed views of the schemas and instances in your project. They let you perform operations to understand how your schema resources relate to other schema resources and what data resides in your instances. The endpoints help you understand the structure of your schema and data. They are particularly helpful for performing deletion operations and knowing as much as possible about the consequences of the deletions. ## Inspecting containers The DMS API exposes an endpoint for inspecting [containers](/api-reference/concepts/20230101/containers) with these operations: * `involvedViews`: Retrieve all views that map a container. * `totalInvolvedViewCount`: Retrieve a count of all views that map a container, optionally including views unavailable to the current user. ## Inspecting instances The DMS API also exposes an endpoint for inspecting [instances](/api-reference/concepts/20230101/instances) with these operations: * `involvedContainers`: Retrieve the containers in which an instance has data. * `involvedViews`: Retrieve the views that map the containers in which an instance has data. # Property graph Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_property_graph Understand how property graphs in CDF use nodes, edges, and properties to model complex industrial systems. In graph modeling, a **property graph** represents a well-defined structure consisting of **nodes** and **edges**, which can be associated with **properties**. These fundamental elements form a simple toolkit for modeling complex systems intuitively and powerfully. ### Nodes **Nodes** serve as fundamental building blocks and can be used to represent anything, for example, real-world objects like *pumps*: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph LR; Pump((Pump)) ``` ### Edges **Edges** denote relationships between nodes. This example adds a *valve* node and an edge to illustrate their connection: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph LR; Pump((Pump)) Valve((Valve)) Pump-- flows-to -->Valve ``` ### Properties Nodes and edges can have **properties** that provide additional context or information. These properties can include **types**, such as `Pump` or `flows-to`, but they can also include other attributes describing the nodes and edges. For example: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph LR; Pump(("Pump
equipmentId: xyz123")) Valve(("Valve
state: OPEN")) Pump--"flows-to
maxPressure: 60"-->Valve ``` # Query features Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_querying Learn how to query your industrial knowledge graph using filters, parameters, recursive edge traversal, and subscriptions to changes. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). The query interface has these endpoints: * `/query` - graph query to retrieve instances matching a query. * `/sync` - graph query to retrieve instances matching a query that have changed since the previously issued cursor. * `/search` - retrieve instances matching full-text search queries. * `/aggregate` - aggregate instance data. A few higher level query endpoints, like `/byids` and `/list`, provide simpler interfaces building on the `/query` endpoint. The query interface supports a wide array of features. * [Advanced filters](#compound-filters) * [Parameterization](#parameters) * [Recursive edge traversal](#graph-traversal) * [Chaining of result sets](#node-result-set-expressions) * [Granular property selection](#selects) * [Subscribing to changes](#syncing---subscribing-to-changes) ## How to define a graph query You can use a query in both the `/query` and the `/sync` endpoints. Queries are composed of: * A `with` section defining result expressions that describe which instances to retrieve. * A `parameters` section with optional parameter substitutions if the query is parameterized. * A `select` section that defines which properties to return as part of the result. The below examples use a pump and valve example, represented as two sets of nodes with edges between them indicating which pumps flow to which valves. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph LR Pump(("Pump
externalId: pump42")) Valve1(("Valve
externalId: valve1")) Valve2(("Valve
externalId: valve2")) PumpValve1Edge("FlowsTo
externalId: 42To1") PumpValve2Edge("FlowsTo
externalId: 42To2") Pump---PumpValve1Edge Pump---PumpValve2Edge PumpValve1Edge-->Valve1 PumpValve2Edge-->Valve2 ``` This query fetches a specific pump as well as the valves it flows to: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: # Define the result expressions pumps: nodes: filter: equals: property: ['node', 'externalId'] value: { 'parameter': 'pumpExternalId' } limit: 1 pump_flows_to_valves: edges: from: pumps maxDistance: 1 direction: outwards filter: equals: property: ['edge', 'type'] value: { 'space': 'types', 'externalId': 'flows-to' } valves: nodes: from: pump_flows_to_valves parameters: # Provide parameter values pumpExternalId: pump42 select: # Define the result sets to return pumps: {} valves: {} ``` The query returns a result similar to this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} items: pumps: - instanceType: node version: 1 space: equipment externalId: pump42 createdTime: 123 lastUpdatedTime: 456 valves: - instanceType: node version: 2 space: equipment externalId: valve1 createdTime: 321 lastUpdatedTime: 654 - instanceType: node version: 3 space: equipment externalId: valve2 createdTime: 213 lastUpdatedTime: 546 nextCursor: pumps: cursorForPumps valves: cursorForValves ``` ## Result expressions Result expressions appear directly below `with` in a query, and define a set of either nodes or edges. You can use the set to return results, as stepping stones to derive other sets from, or both. Result expressions are named and can be chained. A result expression can also define `sort` order and a `limit`. See [sorting](#sorting-and-limiting) for more details. Result expressions can relate to each other via chaining, but they don't have to. You can query for unrelated items in the same query, but you'll generally use different sets to power graph traversals. A set either queries nodes **or** edges, possibly recursively. All fields: * `nodes`: An object to specify a result set of matching nodes. * `edges`: An object to specify a result set of matching edges. * `sort`: A list of sort configurations. * `limit`: How many nodes or edges to return in the result. Default: 100. ### Pagination The max limit you can set for any result expression is 10,000. To retrieve the entire result set, queries accept a `cursor` object mapping result sets to cursors. Pagination cursors in the `nextCursor` object are emitted for each result set, allowing you to page through everything. You can't combine pagination cursors with custom sorts, unless they are backed by a cursorable index. If no pagination cursor is present for a given result expression in the response, there is no more matching data. We do **not** recommend paging through result sets repeatedly using the `/query` endpoint. Instead, use [sync](#syncing---subscribing-to-changes), which is designed to keep clients up to date with changes. ### Node result expressions A `nodes` statement in your result expression makes the set contain nodes. A node result expression can be chained off an optional node or edge result expression. When chaining off another **node result set**, you'll retrieve the nodes pointed to by a direct relation property. The direct relation property is defined using the `through` field. When chaining off an **edge result set**, you'll retrieve the end nodes defined by the edges in the set. * `from`: an optional result expression to chain from. * `filter`: restrict which nodes to return in the result set. * `through`: required when `from` is a node result expression. Specify a direct relation property to traverse. * `direction`: applicable when `through` is defined. Specify which direction to traverse the direct relation. * `chainTo`: applicable when `from` is an edge result expression. Control which side of the edges in `from` to chain to. The behavior depends on the `direction` setting in the `from` result expression: * If `from` follows edges outwards, `direction="outwards"` (default), then `"source"` selects `startNode` and `"destination"` selects `endNode`. * If `from` follows edges inwards, `direction="inwards"`, then `"source"` selects `endNode` and `"destination"` selects `startNode`. ### Edge result expressions An `edges` statement in a result expression makes the set contain edges, and the statement defines the rules the graph traversal will follow. A graph traversal can start from an initial set, defined by `from`, which names another result expression. The graph traversal follows edges in a particular direction, controlled by `direction` (Default: `outwards`.) ``` Alice -is_parent-> Bob Bob -fancies-> Mallory ``` In the above graph, if you follow any edge from `Bob` **outwards** (the default), you'll get the edge `Bob -fancies-> Mallory`. If you follow edges **inwards**, `direction=inwards`, you'll get `Alice -is-parent-> Bob`. The traversal happens breadth-first. See [limitations](#limitations) for more details. A traversal is defined by what edges to follow, what nodes to match, and what nodes to terminate the traversal at. This is controlled by `filter`, `nodeFilter` and `terminationFilter`: * `filter` is a filter on edges. You would typically filter on the property `[edge, type]`, but you can filter on any property on an edge. * `nodeFilter` is a node filter, which the node on the "other" side must match. * With `direction: outwards`, that means the "end node" of the edge must match. * With `direction: inwards`, the "start node" must match. * `terminationFilter` is similar to `nodeFilter`, except if it matches, the traversal ends. A node must also match `nodeFilter` (if any) to steer the traversal to the node to terminate at in the first place. `maxDistance` controls how many hops away from the initial set traversal will go. `maxDistance` defaults to unlimited (but the set must respect its `limit`, defined on the result expression). If `maxDistance` is 1, execution might be faster. If you know there will only be one level, use `maxDistance: 1`. Full options: | Option | | | - | - | | `from` | Result expression to chain from. | | `filter` | Edges traversed must match this filter. | | `nodeFilter` | Nodes on the "other" side of the edge must match this filter. | | `terminationFilter` | Do not traverse beyond nodes matching this filter. | | `maxDistance` | How many levels to traverse. Default unlimited. | | `direction` | Whether to traverse edges pointing out of the initial set, or **into** the initial set. | | `limitEach` | Limit the number of returned edges for each of the source nodes in the result set.

The indicated uniform limit applies to the result set from the referenced from. `limitEach` only has meaning when you also specify `maxDistance=1` and `from`. | | `chainTo` | applicable when `from` is an edge result expression. Control which side of the edges in `from` to chain to. The behavior depends on the `direction` setting in the `from` result expression:

- If `from` follows edges outwards, `direction="outwards"` (default), then `"source"` selects `startNode` and `"destination"` selects `endNode`.

- If `from` follows edges inwards, `direction="inwards"`, then `"source"` selects `endNode` and `"destination"` selects `startNode`. | ## Selects Select configurations appear directly below `select` in a query and specify which data to retrieve for the respective result expression. The configuration specifies a number of sources (views) and a property selector for each. The property selectors define which view properties to emit in the query result. You can have sets whose properties aren't emitted. This is useful if the sets are necessary for chaining, but can be excluded in the final results. Sets that are neither chained nor selected aren't run, but they'll cause a slight query processing overhead. The results are grouped by their respective sets and contain properties that match the property selectors for the set. ## Filters Filters define what a part of the query matches. Filters are tree structures where the operator comes first and then the parameters for that operator. A simple example is the `in` filter: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} in: property: [node, name] values: [movie] ``` If the property `node.name`, which is a text property, is equal to any of the values in the provided list, the node will match. Properties are typed, and what query operators you can use on a property depends on its type. These are the supported filters. ### Equals The `equals` filter performs exact matches ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} equals: property: ['node', 'space'] value: 'someSpace' ``` ### In The `in` filter matches if any of the elements in the list of values match the property. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} in: property: ['node', 'name'] values: ['movie', 'actor'] ``` If the property is an array, it will match if any of the stored values matches any of the provided values. ### ContainsAny The `containsAny` filter matches if *any* of the stored values matches any of the provided values. The property must be an array. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} containsAny: property: ['someSpace', 'someContainer', 'someArrayProperty'] values: ['movie', 'actor'] ``` ### ContainsAll The `containsAll` filter matches if the stored array contains *all* the provided values. The property must be an array. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} containsAll: property: ['someSpace', 'someContainer', 'someArrayProperty'] values: ['value1', 'value2'] ``` ### Range The `range` filter matches if the stored value is within the provided bounds. The property must be a number or timestamp. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} range: property: ['node', 'createdTime'] gte: 2022-08-09T11:39:48+0000 lt: 2024-08-09T11:39:48+0000 ``` The supported parameters are `gt` (greater than), `gte` (greater than or equal to), `lt` (less than), and `lte` (less than or equal to). ### Overlaps The `range` filter matches if the range made up of the two properties `startProperty` and `endProperty` overlap with the provided range. The properties must be numbers or timestamps. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} range: startProperty: ['someSpace', 'someContainer', 'startTime'] endProperty: ['someSpace', 'someContainer', 'endEndTime'] gte: 2022-08-09T11:39:48+0000 lt: 2024-08-09T11:39:48+0000 ``` The supported parameters are `gt` (greater than), `gte` (greater than or equal to), `lt` (less than), and `lte` (less than or equal to). ### Exists The `exists` filter matches if the stored value is not null. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} exists: property: ['node', 'type'] ``` ### Prefix The `prefix` filter matches if the stored value has the provided prefix. The property must be a string or an array. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} prefix: property: ['someSpace', 'someContainer', 'someArray'] value: [1, 2] ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} prefix: property: ['someSpace', 'someContainer', 'someString'] value: 'myPrefix' ``` ### HasData filter A `hasData` filter matches if data is present in a set of **containers** or **views**. There is an implicit `AND` between the containers and views referenced in the filter, and the filter matches only if the node or edge has data in **all** the specified containers and views. When you specify a **container**, the filter matches if the instance has all required properties populated for the container. When you specify a **view**, the filter matches nodes with data in all the containers that the view maps. If the view has an explicit filter defined, that filter is used instead. The following example matches instances that have data in both the container `my_space.my_container` and the view `my_space.my_view.v1`: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} hasData: - type: container space: my_space externalId: my_container - type: view space: my_space externalId: my_view version: v1 ``` For a **view**, the `hasData` filter behaves differently based on the view's configuration: * **No explicit filter on the view**: The `hasData` filter checks that data exists in all containers the view maps from. The filter matches if there is data in `my_space.my_container AND (my_space.c1 AND my_space.c2)`. * **Explicit filter on the view**: The `hasData` filter uses the view's filter instead. The filter matches if there is data in `my_space.my_container` and the data satisfies the conditions defined in `my_space.my_view.v1.filter`. ### Nested Use a `nested` filter to apply the filter on the node pointed to by a direct relation. `scope` specifies the direct relation property you want use as the filtering property. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} nested: scope: ['some', 'directRelation', 'property'] filter: equals: property: ['node', 'name'] value: 'ACME' ``` ### Instance references The `instanceReferences` filter matches instances with fully qualified references (space + externalId) equal to *any* of the provided references. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} instanceReferences: - space: 'my_space' externalId: 'my_instance' - space: 'my_space' externalId: 'my_other_instance' - space: 'another_space' externalId: 'another_instance' ``` ### Compound filters You can combine filters with `and`, `or`, `not`: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} and: - not: in: property: ['node', 'type'] values: ['movie'] - range: property: ['imdb', 'movie', 'released'] gte: { parameter: start } ``` This corresponds to `(NOT node.type in ('movie')) AND imdb.movie.released >= $start`. ### Parameters You can use parameters for values in filters. The parameters are provided as part of the query object, not in the filter itself. This filter is parameterized: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} range: property: [imdb, movie, released] gte: { parameter: start } ``` A query containing this filter will only run if you provide the `start` parameter. The parameter must be compatible with all the types and operators that refer to the parameter. In the above example, the "released" property is a date, and the start parameter must be compatible with the date type. Otherwise, the query will fail, even if the range filter is optional because it's OR-.ed We recommend that you parameterize queries that take user input. This allows you to reuse query plans across queries with a noticable effect on read-heavy workloads. ## Sorting and limiting Sorting and limiting can happen in different places in a query: * In **the result expression**, that is, in the `with` object that defines a node or edge set. * In **the result selection**, that is, under `select` where you can define sets to emit as results. Sorting and limiting the set definitions under `with` transitively affects dependent sets. If you only change the sort order of a `with` expression, dependent sets will not (necessarily) change (based on how the dependent sets are defined). If you, however, put a limit on an expression, all dependent sets will inherit the limit and consequently change. This is also true for sets that aren't emitted via `select`, that is, for sets that are only defined as stepping-stones for other sets. Sorts and limits defined under `select` change the result appearing order for that set only, **not** for dependent sets. This example query would let `some_edges` and `target_nodes` pull from the full amount of nodes in `some_nodes`, even if what's returned as a result for `some_nodes` is capped at 100: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: some_nodes: ... # Omitted. No sorting here. some_edges: from: some_nodes # Omitted. Also no sorting. target_nodes: from: some_edges # ... select: some_nodes: sources: ... sort: - property: ['node', 'space'] direction: descending limit: 100 ``` ### Order of sorting and limiting A limit in an edge traversal applies to when to start traversing, which happens before sorting. Nodes and edges have different sorting and limiting behavior: Nodes sort and limit **simultaneously**, while recursive edge exploration do **limited traversal** ***then* sort**. The top-n set of nodes sorted by some property will be guaranteed to have the top-n of that property for the set. For edges found through traversal, that is, via `edges`, the `limit` applies to how many edges to discover. This may not be all the edges that **could** be discovered in a full traversal. If you start traversing from some node and ask for 100 edges sorted by creation timestamp, the 100 edges discovered before the traversal stops get sorted. The full graph is **not** traversed to find the 100 newest edges in the graph defined by the traversal filters. To sort with a recursive graph traversal, you'll need to specify the sort configuration via `postSort`. An edge traversal with `maxDistance=1` can take a standard `sort` configuration. ## Syncing - subscribing to changes The [`/sync`](/api-reference/concepts/20230101/instances) endpoint lets you subscribe to changes on instances, matching an arbitrary filter. The interface for the `/sync` endpoint is largely similar to the `/query` endpoint but differs in a few ways: * It always returns a value for `nextCursor`. * It returns instances that have changed since the provided cursor. * It returns soft-deleted instances until they have been hard-deleted. These will have a `deletedTime` property set to a non-null timestamp value. Read more about soft deletion [here](/cdf/dm/dm_concepts/dm_ingestion#soft-deletion). * Cursors from `/sync` are valid for a soft deletion grace period, three days by default. * A cursor older than the soft deletion grace period (3 days) may not pick up all changes, especially the deletions. * If the cursor is too old, you must restart the sync or set `allowExpiredCursorsAndAcceptMissedDeletes` to `true` (see #allowing-the-use-of-outdated-sync-cursors). This endpoint is helpful to avoid continuously pulling a full set of instances and putting unnecessary load on the system. Instead, you can use the `/sync` endpoint to subscribe to changes. You'll pull everything once and then keep that state updated incrementally. The Python SDK documentation has [an example](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_modeling.html#example-on-syncing-instances-to-local-sqlite) of how to do this. The endpoint also enables you to sync a subgraph into memory and then use libraries like [NetworkX](https://networkx.org) to perform various specialized graph analysis tasks on live data. You can also use the endpoint to keep specialized dashboards up to date by syncing out all data about a data model. ### Sync modes There are some performance pitfalls when combining sync with filters. The system cannot easily make use of indexes while determining which instances have changed since the sync started. Therefore, the default way of syncing may not be performant if using filters other than a single space filter. Also, sorting is not supported while syncing because any sort would conflict with the order of yielding new changes. Before the sync endpoint yields new changes, it backfills the data that existed before starting the sync. The system needs to scan through all the data to determine which data matches the provided filter. You can use the `mode` parameter to control the backfill phase: * `onePhase` (default): Don't distinguish between the backfill phase and subsequent changes. Use this mode if you sync without a filter or with a single space filter. * `twoPhase`: Split the sync into two stages. This allows the system to take advantage of an index, either the default index or a custom cursorable index, in the backfill stage. This should be used if syncing with e.g. a `hasData` filter. If you want to use a custom cursorable index, you can provide the `backfillSort` parameter which must match the sort of the desired cursorable index. * `noBackfill`: Skip the backfill stage and yield only instances that have changed since the sync started. ### Allowing the use of outdated sync cursors Sync cursors will expire after 3 days. This is because soft-deleted instances are cleaned up after this grace period, so a client using a cursor older than that risks missing deletes. If the `allowExpiredCursorsAndAcceptMissedDeletes` flag is set to true, the API will allow the use of expired cursors. ## Limitations ### Graph traversal Any query that involves a graph traversal will force nested loop-style execution. This will work well enough for traversals limited to a few hundred thousand unique paths. The graph traversal is **breadth**-first, and all possible paths are traversed. This is especially important to keep in mind with traversals across loops. For example, a query that follows all the possible paths of a fully connected graph will likely be terminated due to constraints on either time, memory, or temporary disk usage. ### Timeout errors Queries get canceled with a `408 Request Timeout` error if they take longer than the timeout. If you hit a timeout like this, you must reduce the load or contention or optimize your query. ### Reverse direct relations targeting lists can't be traversed using the graph query endpoint You can't traverse reverse direct relations which target *lists* of direct relations—reverse direct relations with `targetsList=true`—using the `/query` endpoint. You must use the `/search` endpoint instead. Let's say you have an `Activity` with a list of direct relations pointing to nodes which have data in an `Asset` view. The `Asset` view has a reverse direct relation pointing to a set of nodes with data in the `Activity` view. To retrieve the activies related to an asset, you must send a request to the `/search` endpoint where you filter on the `space`+`externalId` combinations of those assets. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} view: space: schema externalId: Activity version: v1 filter: and: - containsAny: property: ['schema', 'Activity/v1', 'assets'] values: - { 'space': 'assets', 'externalId': 'asset1' } - { 'space': 'assets', 'externalId': 'asset2' } - { 'space': 'assets', 'externalId': 'asset3' } ``` ## Graph query examples ### Simple retrieval by `node.type` This is an example query to retrieve all nodes of type `Pump`, and all the properties on the `Pump/v1` view. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: pumps: nodes: filter: equals: property: ['node', 'type'] value: { 'space': 'types', 'externalId': 'pump' } select: pumps: sources: - source: type: view space: equipment externalId: Pump version: v1 properties: ['*'] # All properties ``` ### Retrieve everything in a space This is an example query to retrieve all nodes in the `equipment` space and retrieve the `manufacturer` property on the `Equipment/v1` view. This query is parameterized and can be used for both the `/query` and the `/sync` endpoints. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: pumps: nodes: filter: equals: property: ['node', 'space'] value: { 'parameter': 'space' } select: pumps: sources: - source: type: view space: equipment externalId: Equipment version: v1 properties: [manufacturer] # Only the manufacturer property parameters: space: equipment ``` ### Traverse edges to retrieve workorders related to a set of assets This query retrieves all work orders related to the set of assets with external IDs `asset1`, `asset2`, and `asset3`. It traverses edges of the type `relates-to` that point from work orders to assets. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: assets: nodes: filter: and: - equals: property: ['node', 'space'] value: 'assets' - in: property: ['node', 'externalId'] values: ['asset1', 'asset2', 'asset3'] assets_relate_to_workorders: edges: from: assets # chain off the assets result expression direction: inwards # the edges point from the workorders to the assets filter: equals: property: ['edge', 'type'] value: 'relates-to' workorders: nodes: from: assets_relate_to_workorders filter: # only get active workorders equals: property: ['workorders', '', 'status'] value: 'active' select: workorders: # only select the workorders sources: - source: type: view space: workorders externalId: WorkOrder version: v1 properties: ['*'] # get all the properties ``` ### Traverse direct relations to retrieve the locations of a set of sites Consider a scenario where there are two sets of nodes representing sites and locations, respectively. The sites are connected to the locations using a direct relation property. This query retrieves the set of location nodes related to a set of sites. To traverse direct relation properties performantly, make sure that the property is associated with a b-tree index. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} with: sites: nodes: filter: and: - hasData: - type: container space: sites externalId: Site - in: property: ['node', 'externalId'] values: ['site1', 'site2', 'site3'] locations: nodes: from: sites direction: outwards through: source: # The view mapping the direct relation property to traverse type: view space: sites externalId: Site version: v1 identifier: location # the identifier of the direct relation property to traverse select: locations: sources: - source: type: view space: locations externalId: Location version: v1 properties: ['*'] ``` ### Traverse direct relations inwards traversing_direct_relations_inwards.png ### Traverse direct relations outwards traversing_direct_relations_outwards.png ### Traverse direct relations multiple levels traversing_direct_relations_multiple_levels.png ### Traverse edges single hop traversing_edges_single_hop.png ### Traverse edges recursively traversing_edges_recursive.png ### Traverse edges recursively with a node filter traversing_edges_recursive_with_node_filter.png ### Traverse edges recursively with a termination filter traversing_edges_recursive_with_termination_filter.png ### Traversing edges using the chainTo parameter traversing_edges_chain_to.png # Search features Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_search Learn how to search for text and property values in your knowledge graph using full-text queries, filters, and relevance ranking. Use the data modeling [search endpoint](/api-reference/concepts/20230101/instances) to search for text and property values in the knowledge graph. This article explains tokenization, matching rules, and endpoint differences to help you build efficient search capabilities. For example, you can use the search endpoint to implement **full-text queries** and **multiple field search**, and to **rank results by relevance**. The search endpoint supports: * **Full-text search** across text fields. * **Configurable matching logic** with `AND`/`OR` operators. * **Prefix-based word matching**. * **Boosting exact phrases**. * **Filtering on properties**. * **Converting search results** to a different unit. See the [Query features](/cdf/dm/dm_concepts/dm_querying#filters) article for information about **filtering**. Queries are eventually consistent. Because of indexing delays, it may take a few seconds for new or updated data to become searchable. ## Search process overview The search query is executed through three distinct steps: 1. **Text analysis and tokenization**: the search query text is analyzed and split into tokens. 2. **Instance matching**: the tokens are matched against pre-indexed instance data in your Cognite Data Fusion (CDF) project. 3. **Instance ranking**: matched instances are sorted by their relevance to the query. ## Text analysis and tokenization The instances you ingest are automatically indexed asynchronously. This indexing process includes a **text analysis** step that breaks down the text into smaller components called **tokens**. The system compares the tokens extracted from your search query against the tokens indexed from your instances to determine matches and ranking. The tokenization process includes: * **Splitting text into tokens**: the text is broken into words based on whitespace and punctuation. * **Lower-casing tokens**: all text is converted to lowercase to make searches case-insensitive. ### Tokenization rules The *smarter search results* feature introduces additional tokenization for selected fields if enabled for your CDF project. For more information, see [About smarter search results](/cdf/dm/dm_concepts/dm_smarter_search_results). During tokenization, the following rules apply: * **Letters, numbers, and underscores (`_`)** don't split tokens. For example, `user_name`, `File1`, and `AH12`, remain a single token. * **Periods (`.`), and apostrophes (`'`)**: * Don't split token between letters (for example, `e.g.` and `don't`, remain one token). * Don't split token between numbers (`3.14` remains one token). * Will split tokens in other cases (for example, in `end.`, or between letter and number). * **Commas (`,`)**: * Don't split tokens between two numbers (`1,000` remains one token). * Will split tokens between letters. * **Colons (`:`)**: * Don't split tokens between two letters (`Scale:Linear` remains one token). * Will split tokens between numbers. * **Other non-standard characters, such as double quotation mark (`"`), hyphen (`-`), whitespace**, etc., will split tokens. Refer to the formal specification outlined in [Unicode Standard Annex #29](https://unicode.org/reports/tr29/#Word_Boundaries) for implementation details. ### Tokenization example | Input | Generated tokens | Explanation | | - | - | - | | `Pump_123-ABC` | `pump_123`, `abc` | Hyphen (non-standard character) splits | | `Temperature Sensor` | `temperature`, `sensor` | Whitespace (non-standard character) splits | | `Temperature_Sensor_1` | `temperature_sensor_1` | Underscores (standard characters) don't split | | `John's pump` | `john's`, `pump` | Apostrophe doesn't split the sequence of letters | | `example.com` | `example.com` | Period doesn't split the letter-to-letter sequence | | `system32.exe` | `system32`, `exe` | Period splits the number-to-letter sequence | | `first.last 5.10` | `first.last`, `5.10` | Period doesn't split the letter or number sequences | | `first,last 5,10` | `first`, `last`, `5,10` | Comma splits the letter sequence but not the number sequence | | `first:last 5:10` | `first:last`, `5`, `10` | Colon splits the number sequence but not the letter sequence | | `John's 1st account has 1,000.5$ dollars` | `John's`, `1st`, `account`, `has`, `1,000.5`, `$`, `dollars` | Combined rules | ## Instance matching After tokenization, the search compares your query tokens against indexed tokens from instances in your CDF project. The matching behavior depends on the token's position in your query: * **Standard tokens (exact match):** all tokens in the query **except the last one** require an exact match (case-insensitive) with a token in the instance data. * **Final token (prefix match):** the **last token** is treated as a prefix, allowing for *search-as-you-type* functionality. It matches any word that starts with those characters. ### Example scenario If you search for `pressure valve ma`, the system matches instances based on the following criteria: * **`pressure`** and **`valve`**: require an exact match. * **`ma`**: requires a prefix match (matching `main`, `manifold`, `manual`, etc.). ### Search operators The token matching rules above determine whether individual tokens match a given instance. The **search operator** determines which instances qualify as a match for the entire query. | Operator | Behavior | Example (query: "pressure sensor") | | :- | :- | :- | | **OR** (default) | Returns instances matching **at least one** token. | Matches instances containing "pressure" only, "sensor" only, or both. | | **AND** | Returns only instances matching **all** tokens. | Matches only instances containing both "pressure" AND "sensor". | Effective November 2026, the default search operator will change from `OR` to `AND`. Cognite Flows and Cognite InField already use `AND` as the default operator. To maintain your current search behavior, we recommend explicitly setting the operator in your search queries. ### Limitations on matching Consider these limitations when designing your search: * **No fuzzy or typo matching**: queries require correct spelling and matching prefixes. * **No synonym expansion**: queries are matched literally. Synonyms or abbreviations must appear explicitly in the data. ## Instance ranking When multiple instances match a query, they're ordered by relevance. The following factors determine the ranking: ### Number of matching tokens Instances that match more tokens are ranked higher. This is mainly relevant when using the `OR` operator since the `AND` operator requires all tokens to match. For example, searching for `Heat Exchanger 243 alpha` using the `OR` operator: * Ranks `Heat Exchanger 243` higher (matches three of four tokens). * Ranks `Heat Exchanger` lower (matches two of four tokens). * Ranks `Heat` lowest (matches one of four tokens). ### Phrase matching (exact sequences) Exact phrase matches boost relevance significantly. For example, searching for `heat exchanger`: * Ranks `Heat Exchanger` higher (exact phrase match). * Ranks `Heat For Exchanger` lower (individual tokens matches only). ## Example query matches ### `valve` With `operator: "OR"`: * Matches: `Valve control unit`, `Safety valve unit` * Matches: `Ball-valve` (tokenized as `ball` and `valve`) * Does not match: `Valvoline` (different token) or `Valve's` (tokenized as `valve's`) ### `pressure sensor` With `operator: "OR"`: * Best matches: documents containing both "pressure" and "sensor" * Lower relevance: documents with either "pressure" or "sensor" alone * Example matches: * `High pressure sensor calibration` (matches both tokens) * `Pressure transmitter` (matches only "pressure") * `Temperature sensor` (matches only "sensor") * Does not match: * `Pressured equipment` (pressure is not a prefix query) With `operator: "AND"`: * Requires that **both** "pressure" and "sensor" are present as exact matches. * **Matches**: `High pressure sensor calibration`. * **Does not match**: `Pressure transmitter` (missing "sensor") or `Temperature sensor` (missing "pressure"). ### `compressor fail` With `operator: "OR"`: * Matches: * `Compressor failure log` ("fail" is a prefix of "failure") * `Compressor failing to start` ("fail" is a prefix of "failing") With `operator: "AND"`: * Requires an **exact match** for both "compressor" and "fail". * **Does not match**: `Compressor failure log`, because `fail` is not an exact match for the token `failure`. * **Does not match**: `Compressor failing to start`, because `fail` is not an exact match for the token `failing`. ### `oil temp` With `operator: "OR"`: * Matches: * `Oil temperature readings` ("temp" is a prefix of "temperature") * `Oil temporary storage` ("temp" is a prefix of "temporary") ### `flow meter calibra` With `operator: "OR"`: * Matches: * `Flow meter calibration procedure` (highest rank - all tokens match) * `Flow meter maintenance` (medium rank - two exact tokens match) * `Calibration of temperature meters` (lower rank - only "meter" and "calibra" match) ### `server1.example.com/v2.0` With `operator: "OR"`: * Matches: * `Connect to server1.example.com using v2.0 protocol` (highest rank - all tokens match) * `server2.example.com documentation` (matches example.com) * `API v2.0 reference` (matches version number) * `server1 is down after v2.0.1 upgrade` (matches "server1" and prefix on "v2.0") * Does not match: * `example.net` ("example.net" is preserved as a single token) * `v2` (`v2.0` is preserved as a single token) ## Example search query The search query below filters the results to `Equipment` with `temperatures` between 15 and 25 degrees Celsius, and a `name` or `description` that contains the word "temperature", or a word that starts with "sensor". ```json title="Example search query" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "view": { "type": "view", "space": "testSpace", "externalId": "Equipment", "version": "v1" }, "query": "temperature sensor", "operator": "AND", "instanceType": "node", "properties": ["name", "description"], "targetUnits": [], "filter": { "range": { "property": "temperature", "lt": 25, "gt": 15 } }, "includeTyping": false, "sort": [ { "property": ["externalId"], "direction": "ascending" } ], "limit": 100 } ``` To perform the same query with the **GraphQL** endpoint: ```graphql title="Example search query in GraphQL" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} searchEquipment( query: "temperature sensor", fields: ["name", "description"], filter: { range: { temperature: { lt: 25, gt: 15 } } }, sort: { externalId: ASC } ) { items { externalId name description # Include additional fields here } } ``` ## Filtering differences between the `query` and `search` endpoints Filters work mostly the same for both the `query` and `search` endpoints, but there are a few differences in the handling of empty arrays and prefix arrays. ### Exists filter with empty array | Endpoint | Behavior | Example | | - | - | - | | Query | Empty array counted as existing | `exists([])` → **true** | | Search/Aggregate | Empty array counted as non-existing | `exists([])` → **false** | ### Prefix filter on arrays | Endpoint | Behavior | | - | - | | Query | Checks array prefix sequence (ordered matching). Supports `text[]` and `int[]` arrays. | | Search/Aggregate | Checks each array item separately. Supports only single-value text field prefix filters (no arrays). | #### Examples of prefix filter behavior | Prefix Condition | Query API | Search API | Note | | - | - | - | - | | `"pump"` prefix of `["pump", "valve"]` | ✅ | ✅ | Both APIs match single elements. | | `"pump"` prefix of `["pumping", "valve"]` | ❌ | ✅ | Only Search matches `"pumping"` (element prefix exists). | | `["pump","valve"]` prefix of `["pump","valve","sensor"]` | ✅ | ❌ | Search doesn't support array prefix. | | `"pump"` prefix of `["valve","pump"]` | ❌ | ✅ | Query API checks start sequence. Search any element. | ### Nested filters are only supported for core data model assets Nested filters aren't supported in the search and aggregation endpoints of Cognite data models, except when filtering direct relations to core data model assets. If you need to apply nested filters on properties that are not directly related to core data model assets, use the Query API. #### Supported nested filters The following types and properties support nested filtering in the search and aggregation endpoints: | Core data model type | Core data model property | Supported nested properties | | - | - | - | | [CogniteActivity](/cdf/dm/dm_reference/dm_core_data_model#activity) | assets | `path` | | [CogniteFile](/cdf/dm/dm_reference/dm_core_data_model#file) | assets | `path` | | [CogniteFile](/cdf/dm/dm_reference/dm_core_data_model#file) | category | `code`, `standard`, `standardReference` | | [CogniteTimeSeries](/cdf/dm/dm_reference/dm_core_data_model#timeseries) | assets | `path` | | [CogniteTimeSeries](/cdf/dm/dm_reference/dm_core_data_model#timeseries) | unit | `symbol`, `source`, `sourceReference`, `quantity` | | [CogniteEquipment](/cdf/dm/dm_reference/dm_core_data_model#equipment) | asset | `path` | | [CogniteMaintenance](/cdf/dm/dm_reference/dm_process_industry_data_model#CogniteMaintenanceOrder) | asset | `path` | | [CogniteNotification](/cdf/dm/dm_reference/dm_process_industry_data_model#CogniteNotification) | asset | `path` | | [CogniteOperation](/cdf/dm/dm_reference/dm_process_industry_data_model#CogniteOperation) | asset | `path` | # About smarter search results Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_smarter_search_results Learn about the smarter search results feature, which improves search for tag-like naming structures through advanced tokenization rules. The feature is enabled on a per-project basis. Contact your Cognite account representative to enable the feature for your projects. The *smarter search results* feature enables additional parsing and tokenization to improve searches for *tag-like* naming structures. Tag-like structures usually consist of letters and numbers, optionally delimited by various symbols, such as `PX-14-A-0059` or `AB42/9x14`. ## Affected fields When you enable *smarter search results* for a project, the [additional tokenization rules](#tokenization-rules) apply to the following fields: * `CogniteDescribable.name` * `CogniteDescribable.description` * `CogniteDescribable.tags` * `CogniteDescribable.aliases` ## Tokenization rules For fields affected by *smarter search results*, the following tokenization rules apply in addition to the standard [tokenization rules](/cdf/dm/dm_concepts/dm_search#tokenization-rules): * **All symbols** split tokens. A symbol is any character that is not a letter or a number. For example, `AB-42` splits into two tokens: `AB` and `42`. Similarly, `AB/42` splits into `AB` and `42`. * **Number-letter transitions** split tokens. For example, `AB42` splits into two tokens: `AB` and `42`. Similarly, `42AB` splits into `42` and `AB`. * **Leading zeroes** are removed from tokens. For example, `AB0042` splits into two tokens: `AB` and `42`. This lets you search for `ab42` and find the tag without having to remember how many leading zeroes the tag has. * **Camel case transitions** split tokens. For example, `AbcDef` splits into two tokens: `Abc` and `Def`. # Spaces, instances, direct relations Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_spaces_instances Learn how to construct an industrial knowledge graph using spaces, instances, nodes, edges, and direct relations in CDF. ## Space A **space** can contain both **schemas** and **instances**. It is an efficient resource to help organize your graph. It functions as a namespace, and lets you choose identifiers without interference from other spaces. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph TB subgraph "Space X" X1((A)) --> X2((B)) end subgraph "Space Y" Y2((B)) end ``` Spaces serve as a scope for **governance** and **access control** facilitating a structured approach to data management. You can use CDF user groups and capabilities to define who has access to read from and write to a space. You can, for example, create a data model in a protected space to prevent it from being changed, and then create instances in a space where users can read and write to the data. To delete a space, you have to remove all schema resources assigned to it first. ## Instance The term **instance** is an umbrella term for **nodes** and **edges**. * **Nodes** can represent anything, for example, real-world objects like *pumps*. * **Edges** describe relationships between nodes. Every instance has a set of core properties, and many of the properties apply to both nodes and edges: | Property | Description | Node | Edge | | - | - | - | - | | space | Which space the node/edge belongs to. | x | x | | externalId | The identifier of the node/edge. | x | x | | type | A direct relation pointing to the type node/edge. | x | x | | startNode | A direct relation pointing to the start node. | | x | | endNode | A direct relation pointing to the end node. | | x | | createdTime | When the node/edge was created. | x | x | | lastUpdatedTime | When the node/edge was last updated. | x | x | | version | A number incremented every time the instance is modified. | x | x | When you create an instance, the version is set to 1. Any changes to the instance increase the version by increments of 1. For example, an instance with version 3 indicates that the instance has been updated twice. Update requests that don't update any instance properties don't increment the `version` or `lastUpdatedTime` of the instance. Edges depend on their associated start and end nodes. When a node is deleted, any edges connected to it are also removed. Edges can **span** spaces. They reside in a specific space but can link to nodes in other spaces. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph LR subgraph "Space Y" X1((B)) end subgraph "Space X" E(E) Y2((A)) --- E E --> X1 end ``` ### External IDs Every instance is assigned an **external ID** that must be unique within a space. A fully qualified external ID consists of both the space and the external ID, for example: `{"space": "mySpace", "externalId": "myNode"}`. You can also use the short form: `["mySpace", "myNode"]`. The **maximum length** of an external ID is 255 characters, and **null bytes** are not allowed within the ID. ## Direct relations vs. edges You can express relationships to nodes using **edges** or a special type of property called a **direct relation**. A direct relation is a property holding a reference to a node or list of nodes, and is similar to a *foreign key* in relational models. Edges and direct relations have different characteristics: | | Direct relations | Edges | | - | :-: | :-: | | Can have properties of their own. | No | Yes | | Can be traversed recursively. | No | Yes | | Can restrict which [container](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) the target node must have data in. | Yes | No | | Cheap, supporting a large number of direct relations with minimal overhead. | Yes | No | | Relatively costly, making direct relations a consideration for large quantities, as they count toward instance limits. | No | Yes | | Can enforce that a set of edges form a tree or a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph). | No | Yes | | Can traverse reverse direction of one-to-many relationships.\* | No | Yes | If you have a list of direct relations, you can't traverse the reverse direction of a one-to-many relationship. Instead, you have to do a lookup in the reverse direction. For example, in the Cognite Core model, `CogniteActivity` has a property `assets` which is a list of direct relations to `CogniteAsset`. To find all activities that have a specific asset, you can't create a query that starts from the asset(s). Instead, you have to query all activities and filter on the asset(s) you want. If the relation was modeled as an edge, you could find all activities by traversing the edge from the asset(s). ## Type nodes In your graph, nodes can represent anything from physical entities to abstract concepts like a comment or the *type* of a physical entity. Every instance has a `type` property, a direct relation pointing to the node that defines its intended type. For example, a node representing a physical pump can have the `type` property pointing to a "Pump" node. Or an edge representing a pipe can have the `type` property pointing to a "FlowsTo" node. As type systems grow in size and complexity, the importance of organization and governance increases. To manage type systems effectively, we recommend that you organize them in dedicated spaces within your graph. In the example graph below, the type nodes are in a dedicated `types` space. Depending on the complexity of your type system, you may want to consider organizing them across multiple spaces. You can't delete a type node when it has instances pointing to it. See [Node type support in data modeling](/cdf/integration/guides/transformation/write_sql_queries#write-to-specific-properties-in-data-modeling) to populate the type property of nodes and edges using Transformations. ## Example knowledge graph This example illustrates what a small knowledge graph could look like. It displays nodes, direct relations, and edges to represent a pump directing liquid through pipes into a set of valves and where someone has added a comment about the pump: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph LR; subgraph "space: equipment" Pump(("Pump
externalId: pump42
type: [types, pump]")) Valve1(("Valve
externalId: valve1
type: [types, valve]")) Valve2(("Valve
externalId: valve2
type: [types, valve]")) PumpValve1Edge("FlowsTo
externalId: 42To1
type: [types, flows-to]
startNode: [equipment, pump42]
endNode: [equipment, valve1]") PumpValve2Edge("FlowsTo
externalId: 42To2
type: [types, flows-to]
startNode: [equipment, pump42]
endNode: [equipment, valve2]") Pump---PumpValve1Edge Pump---PumpValve2Edge PumpValve1Edge-->Valve1 PumpValve2Edge-->Valve2 end subgraph "space: users" User(("User
externalId: engineer3
type: [types, user]")) Comment(("Comment
externalId: cmnt144
type: [types, comment]
user: [users, engineer3]
isCommentOn: [equipment, pump42]")) Comment-.->User Comment-.->Pump end subgraph "space: types" direction LR PumpType(("PumpType
externalId: pump")) ValveType(("ValveType
externalId: valve")) CommentType(("CommentType
externalId: comment")) UserType(("UserType
externalId: user")) FlowsToType(("FlowsToType
externalId: flows-to")) Pump-.->PumpType Valve1-.->ValveType Valve2-.->ValveType PumpValve1Edge-.->FlowsToType PumpValve2Edge-.->FlowsToType User-.->UserType Comment-.->CommentType end ``` ### Simplified view Here's a simplified view of the above graph, displaying only the nodes and edges: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} graph TB; subgraph "space: equipment" Pump(("Pump
externalId: pump42
type: [types, pump]")) Valve1(("Valve
externalId: valve1
type: [types, valve]")) Valve2(("Valve
externalId: valve2
type: [types, valve]")) PumpValve1Edge("FlowsTo
externalId: 42To1
type: [types, flows-to]
startNode: [equipment, pump42]
endNode: [equipment, valve1]") PumpValve2Edge("FlowsTo
externalId: 42To2
type: [types, flows-to]
startNode: [equipment, pump42]
endNode: [equipment, valve2]") Pump---PumpValve1Edge Pump---PumpValve2Edge PumpValve1Edge-->Valve1 PumpValve2Edge-->Valve2 end subgraph "space: users" User(("User
externalId: engineer3
type: [types, user]")) Comment(("Comment
externalId: cmnt144
type: [types, comment]
user: [users, engineer3]
isCommentOn: [equipment, pump42]")) end subgraph "space: types" direction LR PumpType(("PumpType
externalId: pump")) ValveType(("ValveType
externalId: valve")) CommentType(("CommentType
externalId: comment")) UserType(("UserType
externalId: user")) FlowsToType(("FlowsToType
externalId: flows-to")) end ``` # System schemas Source: https://docs.cognite.com/cdf/dm/dm_concepts/dm_system_schemas Learn about globally available system schemas prefixed with cdf_ that provide out-of-the-box containers, views, and data models. System schemas are defined in globally available [**spaces**](/cdf/dm/dm_concepts/dm_spaces_instances). They are prefixed with **`cdf_`** and contain containers, views, and data models available out of the box in your Cognite Data Fusion (CDF) project. System schemas and containers implement indexing and constraints for you, and we recommend that you use these system schemas as much as possible. However, it is important you test your data models against your schema with a magnitude of data greater than you expect in production. This to make sure your query patterns fit your schema and data, so you ensure both ingestion and querying are sufficiently performant. The system spaces aren't writable by any user so you can't modify the schemas, or create new instances in them. You can, however, query them like any other space and use the schemas to populate data on instances in your own space. For example, if you want to add `name` and `description` properties to your pump instances, use the `cdf_cdm.CogniteDescribable` container. Any application or client that queries for CogniteDescribable to list an instance, will also show your `name` and `description` without being familiar with the rest of your data model: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph TB classDef ViewClass fill:#8250df,stroke:#24292f,stroke-width:2px,color:#fff classDef ContainerClass fill:#1a7f37,stroke:#24292f,stroke-width:2px,color:#fff subgraph equipment Equipment[["Equipment
manufacturer (string)"]] PropertyManufacturer([manufacturer]) Pump[["Pump
maxPressure (float)"]] PropertyPressure([maxPressure]) BasicPump[["BasicPump
producer (string)
maxPressure (float)
name: (string)
description (string)"]] Equipment:::ContainerClass --has-property--> PropertyManufacturer Pump:::ContainerClass --has-property--> PropertyPressure BasicPump:::ViewClass --maps--> PropertyManufacturer BasicPump --maps--> PropertyPressure end subgraph cdf_cdm CogniteDescribable[["CogniteDescribable
name (string)
description (string)"]] PropertyName([name]) PropertyDescription([description]) CogniteDescribable:::ContainerClass --has-property--> PropertyName CogniteDescribable --has-property--> PropertyDescription BasicPump --maps--> PropertyName BasicPump --maps--> PropertyDescription end ``` # Data modeling concepts Source: https://docs.cognite.com/cdf/dm/dm_concepts/index Learn about the core concepts and features of the Data Modeling Service (DMS) including spaces, instances, containers, views, and property graphs. ## Foundation Understand how property graphs use nodes, edges, and properties to model complex industrial systems. Construct an industrial knowledge graph using spaces, instances, nodes, edges, and direct relations. ## Schema definition Define schemas using containers, views, and data models to bring your own properties into the knowledge graph. Explore globally available system schemas prefixed with cdf\_ that provide out-of-the-box containers, views, and data models. ## Working with data Ingest and remove instances in your graph using apply, patch, replace, and delete operations. Query your industrial knowledge graph using filters, parameters, recursive edge traversal, and subscriptions to changes. Search for text and property values in your knowledge graph using full-text queries, filters, and relevance ranking. Summarize and analyze data using aggregation functions, grouping, and filters. ## High-performance storage Records are individual data objects that represent events, logs, or historical entries. Streams are logical containers that organize your records and define how they behave throughout their lifecycle. ## Management and inspection Control access to graphs using spaces and access control lists (ACLs). Inspect schemas and instances to understand their structure and relationships in your data model. ## Beta features Improved search for tag-like naming structures through advanced tokenization rules. For REST endpoint documentation, see [Data modeling API reference](/api-reference/concepts/20230101/data-modeling). # Data modeling language extension for GraphQL Source: https://docs.cognite.com/cdf/dm/dm_graphql/dm_data_modeling_language Learn about the Data modeling language extension for GraphQL (DML) for managing data models and views. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). The Data Modeling Language extension (DML) offers a simplified way of building data models in CDF based on the [GraphQL Schema Definition Language](https://graphql.org/learn/schema). A **data model** in the context of DML is a collection of related **types** representing the domain specific concepts you want to model for a given use-case. DML functions as an abstraction layer on top of the Data Modeling Service (DMS) API, when you define a data model using DML and publish it, the types and properties in your DML data model are converted into the relevant data models, views, and containers in DMS. The [**data modeling REST API (DMS)**](/api-reference/concepts/20230101/data-models) in combination with the [Cognite Python SDK](/dev/sdks/python) offers more power and flexibility than the DML extension. They're the preferred tools for managing industrial knowledge graphs in Cognite Data Fusion (CDF). ## DML HTTP endpoint Send **`POST`** requests with a JSON GraphQL payload to: ``` https://{cluster}.cognitedata.com/api/v1/projects/{project}/dml/graphql ``` See [Endpoint](/cdf/dm/dm_graphql/dm_graphql_querying#endpoint) in *GraphQL queries* for `{cluster}`, `{project}`, and authentication. Use this `/dml/graphql` URL for DML schema work only. For instance queries and mutations, use the GraphQL URL for your data model on that same page. ## Types **Types** are the basic building block of your data model and represent use-case specific concepts. Each type is associated with one or more **properties** which are each associated with another type. Below is a simple type definition in DML modeling an Equipment type with two properties: **title** and **pages**: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Equipment { title: String pages: Int } ``` ### Extending types via inheritance Using GraphQL inheritance rules, you can create types that extend parent types and inherit all their properties. * Define a type as an **interface** to make it extendable. * Use the `implements` keyword to extend this base type from another type. * Extend multiple interfaces by separating them with an ampersand (`&`). In most implementations of the GraphQL Schema Definition Language, interfaces are treated differently from types; they cannot be queried directly and serve only as blueprints to be implemented by other types. In DML, interfaces function as both types and interfaces. This means that when you change a type to an interface, the only functional difference is that it can now serve as a parent type for others. You can still query the interface via the CDF APIs as if it were still a standalone type in your data model. Below is an example of inheritance in DML: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} interface Equipment { name: String installationDate: Timestamp } type Pipe implements Equipment { name: String installationDate: Timestamp diameter: Float length: Float } ``` In this example, the parent type (`Equipment`) is implemented by the `Pipe` type, which inherits all parent properties and adds specific properties relevant only to the child type. There are some caveats with fields that are imported from inheritance, learn more in the [advanced section](#advanced-mapping-and-importing-fields). ## Properties A type consists of a collection of properties. Each property can be a `scalar`, an `enum`, or a reference to a `custom type`. When a property references a custom type, it represents a `relation` between instances in your data model. ### List properties By default a property can only contain a single value: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Book { # A single book can only have a single tag tag: String } ``` To make a list property that can contain multiple values, add brackets (`[]`) around the type of the property: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Book { # A single book can have multiple tags tag: [String] } ``` #### Limiting number of values on list properties By default list properties can have maximum of between 100 and 1000 values [depending on the property type](/cdf/dm/dm_reference/dm_limits_and_restrictions#list-size-limits). To override the default maximum values on a list property, use the `@limits` directive: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Book { # A single book can have up to 3 tags tag: [String] @limits(maxListSize: 3) } ``` ### Scalar properties Scalar properties represent primitive data. | DML type name | DMS name | Description | | - | - | - | | `String` | text | | | `Int` or `Int32` | int32 | 32-bit integer number | | `Int64` | int64 | 64-bit integer number | | `Float` or `Float64` | float64 | 64-bit number encoded in IEEE 754 | | `Float32` | float32 | 32-bit number encoded in IEEE 754 | | `Timestamp` | timestamp | Timestamp encoded in ISO 8601 | | `JSONObject` | json | JSON encoded data | | `Date` | date | Date encoded in ISO 8601, but without time | | `Boolean` | boolean | | ### Enum properties Enum properties are special scalars that must match one of several predefined, distinct values. To create one, declare an enum and then use that enum as the type of another property: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} enum StatusEnum { ACTIVE INACTIVE } type Equipment { name: String! status: StatusEnum } ``` ### CDF resource type properties Properties can also reference non-data-modeling resource types in CDF. The value of these properties is a string reference to the external ID of the underlying resource. | GraphQL type | CDF resource | | - | - | | File | [files](https://developer.cognite.com/dev/concepts/resource_types/files) | | TimeSeries | [timeseries](https://developer.cognite.com/dev/concepts/resource_types/timeseries) | | Sequence | [sequences](https://developer.cognite.com/dev/concepts/resource_types/sequences) | Using these native types enables specialized querying capabilities via the GraphQL API and some of our SDKs. Below is an example of a type having CDF native resource properties: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Equipment { name: String documentation: File temperature: TimeSeries voltage: TimeSeries performanceCurve: Sequence } ``` ### Relation properties **Relations** allow types to reference other types, forming connections within your data model. These can point to single or multiple target instances. You create relation properties in DML by defining a property that uses a custom type as its type: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Author { name: String birthDate: Timestamp } type Book { name: String isbn: String author: Author } ``` This example models a type representing a book with a relation connecting it to a single author. In DMS there are three types of relation properties: 1. **Direct relations:** A container property storing a reference to another node. 2. **Edges:** A connection property that can be used to describe one-to-many or many-to-many relationships. 3. **Reverse direct relation:** A connection property representing a direct relation in another type going in the opposite direction. You can read more about these in the [direct relations vs. edges](/cdf/dm/dm_concepts/dm_spaces_instances#direct-relations-vs-edges) and the [reverse direct relations](/cdf/dm/dm_concepts/dm_containers_views_datamodels#reverse-direct-relation) sections in the data modeling documentation. The type of relation property used for a property in DML is determined by several factors described below. #### Direct relations A relation with a single target instance defaults to a **direct relation**: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # This will become a direct relation property in DMS documentation: Document } ``` To model a **list** of references using direct relations, you must add the `@directRelation` directive. Without this, list property will default to being modelled as an **edge**: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # This will become a direct relation property in DMS documentation: [Document] @directRelation } ``` #### Reverse direct relations A **reverse direct relation** looks "backward" through an existing direct relation on the target type. You must specify the `throughProperty` on the target type for the graph traversal. ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Document { # The "throughProperty" must match the name of the direct relation on the target type referencingAssets: Asset @reverseDirectRelation(throughProperty: "documentation") } type Asset { documentation: Document } ``` #### Edges Since single-target relations default to direct relations, add the `@relation` directive to define a single-target edge relation: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # This will become an edge connection property in DMS documentation: Document @relation } ``` A multiple target relation property defaults to edge so in this case no `@relation` directive is necessary to model it as an edge: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # This will become an edge connection property in DMS documentation: [Document] } ``` ##### Properties on edges Edges can possess their own properties. To define these on an edge relation, create a type annotated with the `@edge` directive and link it using `edgeSource` in the `@relation` directive: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # The @edge directive specifies that this type represents the properties of an edge relation property type PowerCable @edge { ratedVoltage: Float installationDate: Timestamp } type Asset { # Represents a connection between two assets via a power cable edge that has its own properties connectedTo: Asset @relation(edgeSource: "PowerCable") } ``` ##### Overriding edge type The **edge type** used to identify an instance of an edge defaults to `{your_typename}.{your_fieldname}`. However, you can override this behavior and manually specify the edge type using the `type` field within the `@relation` directive. ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # Instances of this edge will be identified by the `power_cable` node in the `my_types` space connectedTo: Asset @relation(type: { externalId: "power_cable", space: "my_types" }) } ``` ##### Overriding edge direction The relational direction for edges defined in DML is **OUTWARDS** by default. To create a relation that points inwards (referencing incoming edges), add the `direction` argument to `INWARDS` on the `@relation` directive. ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Asset { # All power connections from this asset to another asset (Default: OUTWARDS) connectedTo: Asset @relation(type: { externalId: "power_cable", space: "type_space" }, direction: OUTWARDS) # All power connections to this asset from some other asset connectedFrom: Asset @relation(type: { externalId: "power_cable", space: "type_space" }, direction: INWARDS) } ``` ### Descriptions GraphQL descriptions are supported and will be converted to the corresponding description property on view, container and property definitions. To set the `name` of a view, container, or `field`, start a line in the description with `@name` followed by the name value. Example of GraphQL descriptions: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} """ This is a common interface for common models @name Common View """ interface Common { name: String } type Equipment implements Common { """ This is the description for the property Equipment.name @name Name """ name: String } ``` ## Specification of directives The following is a list of the directive definitions, with descriptions. ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} """ Specifies that a type is a view type. * space: Overrides the space, which by default is the same as the data model. * version: Overrides the version of the view, which by default is the same as the data model version. * rawFilter: Sets a filter on the view, accepts a JSON string. """ directive @view( space: String version: String rawFilter: JSON ) on OBJECT | INTERFACE """ Specifies that a type is an edge type. The view can then be used to expand an edge node with a node instance with properties as specified in the view with the @edge directive. The edge should be annotated with `@relation(edgeSource: "name_of_edge_view")`. """ directive @edge ON OBJECT | INTERFACE """ Specifies that a type is importing another view and its backing container. Currently, this annotation must be combined with @view to specify the view and view version to import. In order to extend the view in your data model, specify it as an interface and then create a new data type that implements the interface and adds additional properties. In the future, this directive will be extended to simplify importing. """ directive @import ON OBJECT | INTERFACE """ Overrides the mapping of a field. Can only be used in a view type and can not be used on imported fields. * space: Overrides the space, which by default is the same as the data model space. * container: Overrides the container externalId, which by default is the same as the externalId of the view postfixed with 'Container'. * property: Overrides the container property identifier being mapped. """ directive @mapping( space: String container: String property: String ) on FIELD_DEFINITION """ Defines the relation field's details * type: The reference to the edge type this relation is mapped to. * name: Overrides the name property of the relation definition. This is merely metadata, and should not be confused with the property identifier! * edgeSource: The name of the edge view to use for expanding the edge node with a node instance. * direction: The direction to follow the edges filtered by 'type'. * type: Specifies the edge type, namespaced by 'space', where the 'externalId' corresponds to the edge type name. """ directive @relation( type: _DirectRelationRef name: String edgeSource: String direction: _RelationDirection ) on FIELD_DEFINITION """ Configures the backing container of the view. It can be used to configure indexes or constraints for the properties of a container. """ directive @container( constraints: [_ConstraintDefinition!] indexes: [_IndexDefinition!] ) on OBJECT | INTERFACE """ Sets the default value of a field. * value: Specifies the default value. The input is always in string format, however all dms supported default value types are also supported. """ directive @default( value: String ) on FIELD_DEFINITION """ Sets the unit on a field. * externalId: specifies a unit by using an external ID value. `externalId` can only be assigned to the types `Float`, `Float32`, `Float64`, or to an array of listed floating types and must match a unit in the unit catalog. * sourceUnit: a free text field to store arbitrary unit information like unit name in a source system. """ directive @unit( externalId: String, sourceUnit: String ) on FIELD_DEFINITION """ Represents an index definition. * identifier: A unique identifier for the index * indexType: Is the type of index * fields: List of field names to define the index across. Only supported for non-inherited or mapped fields. The order of the fields matters. * cursorable: With cursorable index we can efficiently query with custom sort options, and queries will emit cursors that can be used to paginate through the results. Only supported for BTREE indexType. """ input _IndexDefinition { identifier: String! indexType: _IndexType fields: [String!]! cursorable: Boolean } """ Represents a constraint definition. The constraint definition can either be an uniqueness constraint or requires. In case 'constraintType' is set to 'REQUIRES', the 'require' argument must be set. And if it's set to 'UNIQUENESS', the 'fields' argument must be set. * identifier: A unique identifier for the constraint * constraintType: Is the type of constraint * fields: List of field names to define the unique constraint across. Only supported for non-inherited or mapped fields. The order of the fields matters. * require: Specify a container (by space and externalId), which is required to exist on the node, if the current container is to be used. """ input _ConstraintDefinition { identifier: String! constraintType: _ConstraintType require: _DirectRelationRef fields: [String!] } """ Overrides the constraints on the possible values of a property * maxTextSize: If set overrides the max number of characters allowed on a string property * maxListSize: If set overrides the max number of values on a list property """ directive @limits( maxTextSize: Int, maxListSize: Int ) on FIELD_DEFINITION enum _ConstraintType { UNIQUENESS REQUIRES } enum _IndexType { BTREE } input _DirectRelationRef { space: String! externalId: String! } enum _RelationDirection { INWARDS OUTWARDS } ``` ## Advanced: Types and their relation to DMS views and containers Types in the DML data model are abstractions of DMS schema resources (views, containers, and data models). When you publish a DML data model in your CDF project, it's persisted in DMS as a collection of schema objects (containers, views, and data models). Similarly, when you open an existing data model in DML, the DML schema is regenerated from the published DMS schema objects. ### Containers and views Each type in a DML data model has an equivalent view and container. The identifiers for these resources are determined by the following rules: **Container identification** (by space and external ID): * **External ID**: Uses the `@container` directive's external ID if present, otherwise defaults to the type name. * **Space**: Uses the `@container` directive's space if present, otherwise defaults to the data model's space. **View identification** (by space, external ID, and version): * **External ID**: Uses the `@view` directive's external ID if present, otherwise defaults to the type name. * **Space**: Uses the `@view` directive's space if present, otherwise defaults to the data model's space. * **Version**: Uses the `@view` directive's version if present, otherwise generates a version identifier based on the type definition (ensuring any changes to the type definition result in a new view version). ### View and container properties Once a type has been mapped to a view and container, its properties are also mapped to equivalent view and container properties. Each type property always has an equivalent view property in the view mapped to the type. However, which container property the view property maps to is determined by the following rules (in order of precedence): 1. **Explicit mapping**: If the type property has a `@mapping` directive, the directive determines which container property to use. 2. **Inherited mapping**: If the property is inherited from a parent type, the view property maps to the same container property as the equivalent property in the parent type. 3. **Default mapping**: Otherwise, the view property maps to a container property with the same name in the container mapped to the type. When you use the `@mapping` directive, the system doesn't auto-create the referenced container property if it doesn't exist. Only use `@mapping` to reference container properties that already exist. ### Example DML to DMS mapping ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} interface Equipment { name: String } type Valve implements Equipment @view(externalId: "Valve2") { name: String diameter: Number material: String } ``` When you publish this DML data model, it creates the following DMS resources: **Equipment interface:** * **View**: Created with external ID `Equipment` in the data model's space (no `@view` directive, so it uses the type name) * **Container**: Created with external ID `Equipment` in the data model's space (no `@container` directive, so it uses the type name) * **Property `name`**: * View property `name` in the `Equipment` view * Maps to container property `name` in the `Equipment` container (no `@mapping` directive and not inherited, so a new container property is created with the same name) **Valve type:** * **View**: Created with external ID `Valve2` in the data model's space (explicitly set via `@view` directive) * **Container**: Created with external ID `Valve` in the data model's space (no `@container` directive, so it uses the type name) * **Property `name`**: * View property `name` in the `Valve2` view * Maps to container property `name` in the `Equipment` container (inherited from the parent `Equipment` interface, so it reuses the same container property as the parent) * **Property `diameter`**: * View property `diameter` in the `Valve2` view * Maps to container property `diameter` in the `Valve` container (no `@mapping` directive and not inherited, so a new container property is created) * **Property `material`**: * View property `material` in the `Valve2` view * Maps to container property `material` in the `Valve` container (no `@mapping` directive and not inherited, so a new container property is created) The `Valve` type's `name` property references the `Equipment` container rather than its own `Valve` container. This is because inherited properties always map to the same container property as their parent type. ## Spaces and naming collisions When you create multiple data models in **the same space** with identical type names, they will interfere with each other and cause errors. Consider this scenario where you have two data models in the same space: **Data model A:** ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Foo { bar: String! } ``` **Data model B (causes collision):** ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Foo { foo: String } ``` When you try to publish data model B, you'll see an error like this: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Error: could not update data model An error has occurred. Data model was not published. [Line: 28] The view (space: 'test', externalId: 'Foo', version: '9317f4591a0e6c') is not mapping a required property 'bar' of the referenced container (space: 'test', externalId: 'Foo'). This error is likely caused by deleting a field which was non-nullable. Please add this field back or add '@readonly' to 'Foo' ``` This happens because these two data models will have view properties that map to the same container property but with different requiredness. To prevent naming conflicts: * **Use one data model per space** - This eliminates the possibility of collisions. * **Use unique type names** - If you need multiple data models in the same space, ensure all type names are unique across models. # GraphQL mutations Source: https://docs.cognite.com/cdf/dm/dm_graphql/dm_graphql_mutations Learn how to create and update instances using GraphQL mutations including upsert and delete operations. You can create and update instances using GraphQL. The mutation capabilities available from the GraphQL API represent a subset of the capabilities of the [ingestion](/cdf/dm/dm_concepts/dm_ingestion) REST API endpoints. The GraphQL interface for data modeling doesn't anticipate receiving significant data ingestion traffic. For that, please use the DMS REST API, or the CDF Transformations service. For a data model like the example below, mutations are generated for each type, except types annotated with the `@edge` directive: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Movie { title: String actors: [Person] @relation( edgeSource: "ActedIn" type: { space: "imdb", externalId: "HAS_ACTOR" } direction: OUTWARDS ) director: Person } type Person { name: String! } type ActedIn @edge { role: String compensation: Float metadata: JSONObject lines: [String] } ``` ## `upsert` mutations The mutations receive a list of instances you want to create for a specific type. For example: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} mutation CreateMovie { upsertMovie( items: [ { space: "imdb", externalId: "the-prestige", title: "The Prestige" } { space: "imdb", externalId: "the-titanic", title: "The Titanic" } ] ) { externalId space createdTime } } ``` You can extend the mutations to create or upsert nested types. When you upsert nested types, the input has a structure with `node` and `edge`. Direct relations only require `node`. `node` includes the fields for the target type, while `edge` include the fields for the connecting edge. The edge could contain properties, if an edge type is defined in the model, or could be simply `space` and `externalId`. It's similar for direct relations, though only `node` should be included, no `edge`. A more complex example could look like this: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} mutation CreateMoviesWithActors { upsertMovie( items: [ { space: "imdb" externalId: "the-prestige" title: "The Prestige" director: { node: { space: "imdb" externalId: "christopher-nolan" name: "Christopher Nolan" } } actors: [ { edge: { space: "imdb" externalId: "prestige-christian-bale" role: "Alfred Borden" compensation: 10000000 lines: ["Are you watching closely?"] } node: { space: "imdb" externalId: "christian-bale" name: "Christian Bale" } } { edge: { space: "imdb" externalId: "prestige-hugh-jackman" role: "Robert Angier" compensation: 10000000 lines: ["No one cares about the man in the box"] } node: { space: "imdb" externalId: "hugh-jackman" name: "Hugh Jackman" } } ] } ] ) { externalId space createdTime lastUpdatedTime } } ``` You can remove a direct relationship by setting its property to `null`. For example, to remove a director from a movie, set the director property to `{... director:null, ...}`. * **Properties** * `externalId` * `space` * `createdTime` * `lastUpdatedTime` * `wasModified`: a `Boolean` to specify whether an existing instance was modified or a new instance was created, or an instance was unchanged. * `instanceType`: `node` or `edge` * `version` * **Parameters** * `items`: a list of the instances to upsert. This has the shape of the type being upserted. Any nested items have the `node` field to contain the properties of the target type. Relations also have the `edge` field to contain edge properties. # GraphQL queries Source: https://docs.cognite.com/cdf/dm/dm_graphql/dm_graphql_querying Learn how to query instances using GraphQL including list, get, search, and aggregate operations. ## Overview You can query the instances with [GraphQL](https://graphql.org/learn). The query capabilities of the GraphQL API interface represent a subset of the capabilities available from the DMS [query](/cdf/dm/dm_concepts/dm_querying) endpoints. A query has these key sections: * A **query intent** `query MyQueryName {` (or just `{` as shorthand). * The **query to run**. This can be one of four types of queries (with an optional alias, for example, `myAlias: listActor`): * `list`, for example, `listActor` lists based on filter parameters. * `getById`, for example, `getActorById` lists based on filter parameters. * `search`, for example, `searchActor` searches based on a query parameter. * `aggregate`, for example, `aggregateActor` aggregates (count, average etc.) based on a field. See the [query types](#query-types) section for more details. * The **query parameters**. For example, `filter: {...}` or `limit: 1000`. * The **properties to return** at a specified depth. GraphQL `list` and `getById` queries aren't designed for retrieving large datasets. Refer to [GraphQL list and getbyid queries with subqueries](#graphql-list-and-getbyid-queries-with-subqueries) for more information. ## Endpoint Send **`POST`** requests with a JSON body in the form `{ "query": "", "variables": { ... } }` to: ``` https://{cluster}.cognitedata.com/api/v1/projects/{project}/userapis/spaces/{spaceExternalId}/datamodels/{dataModelExternalId}/versions/{dataModelVersion}/graphql ``` `{cluster}` is your CDF API hostname (for example `api` or a regional cluster such as `westeurope-1`). See [IP addresses for the API](/dev/use_the_API#ip-addresses-for-the-api) for more information. `{project}` is your project name. `{spaceExternalId}`, `{dataModelExternalId}`, and `{dataModelVersion}` identify the published data model. [Authenticate](/cdf/access/concepts/authentication_flows_oidc) with a bearer token. DML **schema** operations use `.../projects/{project}/dml/graphql` instead. See [DML HTTP endpoint](/cdf/dm/dm_graphql/dm_data_modeling_language#dml-http-endpoint) for details. ## GraphQL example ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query myQuery { listCountry { items { name demographics { populationSize growthRate } deaths { datapoints(limit: 100) { timestamp value } } } pageInfo { endCursor } } listDemographics( filter: { and: [{ populationSize: { gte: 2 } }, { populationSize: { lte: 10 } }] } ) { items { populationSize growthRate metadata { key value } } } } ``` Where: * `myQuery` declares the **query intent**. * `listCountry` and `listDemographics` are the **queries to run**. * The **parameters** are `Country`:`100` for the `deaths` time series data points, and the `Demographics`: filter of `populationSize` between 2 and 10. * The **properties to return**, for `Country`: `items` -> `name`, `demographic` -> `populationSize`, `growthRate`, etc. For more information, see the [GraphQL documentation](https://graphql.org/learn). ## Query types Use these basic query types: "list", "getById", "search" and "aggregate". ### `list` queries `list` lets you filter and sort data quickly. For `list` you can use these properties and parameters: * **Properties** * `items`: specify the properties to return data for * `pageInfo`: details about the current set of returned items and cursors used to paginate to the next page of data. * **Parameters** * `filter`: the filter you want to use based on the `properties` in your graph. You can use `and`, `or`, and `not` operators to create complex queries. * `sort`: the ordering of the returned data you want. Specify an array of one or more `properties` to sort the data on, and whether the data should be sorted in `ASC` or `DESC` order. * `after`: where in the result set to continue from, using the `endCursor` value from the previous query. * `first`: the maximum number of items to return. To fetch more items use a new query, and supply the value from the previous query's `endCursor` parameter. (**Default maximum**: 1000 instances) #### Pagination Pagination is only supported at the top level when using `list` queries. For a query like the one below, cursors will only be returned to paginate through the data returned from `Movies`, but not from `Actors`. If you need to paginate through nested types, for `actors` in this example, use the `getById` queries instead: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query ListMovies { listMovie { items { title actors { name } } pageInfo { endCursor } } } ``` ### `getById` queries Use `getById` to fetch a specific item by its `externalId`. For `getById`, you can use similar properties and parameters as [`list`](#list-queries) queries. Pagination for nested items is available when using `getById`. * **Properties** * `items`: specify which properties to return. ### `search` queries Use `search` to search and query data quickly. For `search` you can use these properties and parameters: * **Properties** - The same as for the [`list`](#list-queries) queries. * **Parameters** * `filter`: the filter you want to provide based on the `properties` in your data model. You can use an array of `and`, `or`, and `not` to create more complex queries. * `fields`: the properties you want to search on. This can be an array of `properties`. * `query`: the actual search queries. Wildcards are supported. * `first`: the maximum number of items to return. To fetch more items, use a new query using the `endCursor` parameter.(**Default**: 1000) ### `aggregate` queries Use `aggregate` to aggregate data and compute rollups on the instances in a type. For `aggregate`, you can use these properties and parameters: * **Properties** * `group`: depending on the `groupBy` parameter, this indicates the group. The group is the name of each aggregation bucket based on the value(s) of the field(s) used in the `groupBy` parameter. * `avg`: for number properties, you can ask for the average. * `count`: for any type of properties, you can ask for the total count. * `histogram(interval: )`: for number properties, you can ask for the histogram for an interval. * `max`: for number properties, you can ask for the maximum value. * `min`: for number properties, you can ask for the minimum value. * `sum`: for number properties, you can ask for the total sum. * `pageInfo`: the details about the current set of returned items. The existence of `endCursor` indicates that there is another page of data. * **Parameters** * `filter`: perform the aggregation on instances matching a logical expression. This filter is applied to the `fields` in your data type as described below, and you can use an array of `and`, `or`, and `not` to create more complex queries. For text field searches other than prefix matching, use `query`. * `query`: applies a full-text search to text `fields` in the type and performs the aggregation on instances matching the query and filter. The query is, by default, applied to all text fields in the type, but can be constrained to a subset of the `field`s using the `fields` parameter. * `fields`: the properties or fields you want to search in with the `query` search. This can be an array of `field`s. * `groupBy`: the `field`(s) to group by (no relationship allowed). * `after`: the `nextCursor` from the previous query, for pagination. ### Filter specifications | | Field | Filter | How to use | | - | - | - | - | | Top level filters | Any | `and` | `{and: [{...},{...}]}` = if everything is true. | | | | `or` | `{or: [{...},{...}]}` = if at least one is true. | | | | `not` | `{not: {...}}` = if false. | | Primitives | Any | `eq` | `{eq: ...}` | | | | `in` | `{in: ["..."]}` = if any of term. | | | Nullable | `isNull` | `{isNull: True}` = if the field is empty. | | | String | `prefix` | `{prefix: "..."}` = starts with. | | | Int/Int64/Float/Timestamp | `lt` | `{lt: ...}` = less than. | | | | `gt` | `{gt: ...}` = greater than. | | | | `lte` | `{lte: ...}` = less than or equals. | | | | `gte` | `{gte: ...}` = greater than or equals. | | | TimeSeries | ❌ | Not supported. | | User defined type | | ❌ | Not supported. | | Lists(\[XXX]) | | ❌ | Not supported. | ### Unit conversion Fields that are enabled for unit conversion will contain `targetUnit` and `targetUnitSystem` attributes. * `targetUnit`: specifies the external ID of the unit that the retrieved data will be converted to. * `targetUnitSystem`: specifies the unit by the unit system the retrieved data will be converted to. A query can convert at most 10 properties. Each field that sets `targetUnit` or `targetUnitSystem` counts toward this limit. If you exceed it, the request fails with `targetUnits size must be between 1 and 10`. See [Units](/dev/concepts/resource_types/units/units) for more information and examples, and [Limits and restrictions](/cdf/dm/dm_reference/dm_limits_and_restrictions#instance-query-limits) for the constraint. ### Unit information `unitInfo` object contains unit information, `unitExternalId`, `sourceUnit` for unit enabled fields. See [Units](/dev/concepts/resource_types/units/units) for more information and examples. ### Pagination in `list` and `getById` queries GraphQL `list` and `getById` queries support cursoring. In `list` queries, you can cursor only at the top-level. In `getById` queries, you can cursor only at the first sub-query level. The `hasNextPage` field in `pageInfo` will under certain conditions be `true` even when there is not more data. Specifically if the total number of items is divisible by `first`. You should therefore always exhaust cursors until no items are returned, and not rely upon `hasNextPage`. ### GraphQL `list` and `getById` queries with subqueries GraphQL `list` and `getById` queries include subqueries on fields backed by direct relations or edges. When querying these fields, a maximum result set size of **10,000** items applies. This limit refers to the *total* number of items returned by a subquery, not per item in the parent subquery. For example: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query ListActors { listActors(first: 100) { name actedTogether(first: 10) { items { name actedTogether(first: 15) { items { name } } } } } } ``` Here, `Actors` is a type with fields `name` and `actedTogether`, where `actedTogether` is an edge property. * The root `listActors` query returns up to 100 items. * The first nested `actedTogether` subquery has a limit of 10 items. * The second nested `actedTogether` subquery has a limit of 15 items. If the dataset exceeds the subquery limits, the root query will still return 100 items, each item containing 10 `actedTogether` items, and each of those containing 15 more `actedTogether` items. The total number of items returned in the `actedTogether` subquery would be: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} 100 (root) × 10 (actedTogether level 1) × 15 (actedTogether level 2) = 15,000 items ``` This exceeds the maximum allowed result set size of 10,000 items. When the result set size limit is exceeded, any additional items beyond the limit will be excluded from the response. ### Risks of using `first` in GraphQL subqueries Cognite supports using the `first` parameter at the top-level in a "list" query and at the first subquery level of a "getById" query. The parameter limits the number of returned items, and you can use it in combination with a sort to pick the first/last items of a query. For other subqueries, `first` is allowed for legacy support, but shouldn't be used. * In subqueries targeting a list of direct relations, or a reverse direct relation, `first` is allowed but not implemented. Supplying the parameter has no effect on the result. * In subqueries targeting edges, `first` is allowed and implemented, but isn't compatible with `sort`. The limit imposed by `first` is applied **before** the items are sorted. #### Example of using `first` in a list query In a `list` query, `first` is correctly applied if it's used at the top-level of the query, but not if it follows a list of direct relations or a reverse direct relation in a subquery. For example: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query MyQuery { listMovie(first: 1) { items { producers(first: 1) { items { name } } } } } ``` In this example, "Movie" is a type with a "producers" field that's backed by either a list of direct relations or a reverse direct relation. The result of the top level list query "listMovie" will be correctly limited to a single result. The "producers" subquery, however, won't be limited. Example output: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "listMovie": { "items": [ { "producers": { "items": [ { "name": "producer_1" }, { "name": "producer_0" } ] } } ] } } ``` #### Example of using `first` in a getById query In a `getById` query, you can use `first` at the first subquery level. For example: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query MyQuery { getMovieById(instances: { space: "movies", externalId: "movie_0" }) { items { producers(first: 1) { items { externalId } } } } } ``` Where "producers" could be a subquery targeting either a list of direct relations, a reverse direct relation, or an edge connection property. Example output: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "getMovieById": { "items": [ { "producers": { "items": [ { "externalId": "producer_0" } ] } } ] } } ``` However, you can't use `first` in further nested subqueries. For example, the following query, where we use `first` for a nested subquery, "movies", won't work as expected: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query MyQuery { getMovieById(instances: { space: "movies", externalId: "movie_0" }) { items { producers(first: 1) { items { externalId movies(first: 1) { items { externalId } } } } } } } ``` The result of the "movies" subquery won't be limited to a single item. Example output: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "getMovieById": { "items": [ { "producers": { "items": [ { "externalId": "producer_0", "movies": { "items": [ { "externalId": "movie_0" }, { "externalId": "movie_1" } ] } } ] } } ] } } ``` # Autogenerated GraphQL types Source: https://docs.cognite.com/cdf/dm/dm_graphql/dm_graphql_types Learn how CDF automatically generates GraphQL types for data models and how to use them for queries and mutations. When you have defined an industrial knowledge graph with the data modeling REST API, use the GraphQL interface to [**query**](/cdf/dm/dm_graphql/dm_graphql_querying) and [**mutate**](/cdf/dm/dm_graphql/dm_graphql_mutations) data in the instances mapped by the views in your graph. The capabilities available via the GraphQL interface to data modeling represent a subset of the available capabilities from the DMS [query](/cdf/dm/dm_concepts/dm_querying) and [ingestion](/cdf/dm/dm_concepts/dm_ingestion) endpoints. CDF will automatically generate GraphQL types for the collection of views making up a DMS data model. An example generated type: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Movie { "name of the movie" # inlined property description name: String! # an exclamation mark denotes a required property actors: [Actor] # An edge connection property. Is resolved by the GraphQL interface producers: [Producer] @directRelation # A direct relation property. Can be resolved by the GraphQL interface. director: Person # A direct relation property. Can be resolved by the GraphQL interface. watchedIt: Boolean } ``` # Using GraphQL with data modeling Source: https://docs.cognite.com/cdf/dm/dm_graphql/index Learn how to use GraphQL to query, mutate, and define data models in Cognite Data Fusion. Use GraphQL to interact with your data models in Cognite Data Fusion (CDF). GraphQL provides a flexible query language for reading and writing data, as well as a data modeling language for defining schemas. Understand how CDF automatically generates GraphQL types for data models and how to use them for queries and mutations. Query instances using GraphQL including list, get, search, and aggregate operations. Create and update instances using GraphQL mutations including upsert and delete operations. Use the Data modeling language extension for GraphQL (DML) to manage data models and views. # Optimizing data models for AI Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_best_practices_ai_search Learn how to model data to support natural language queries and enable Atlas AI agents to be reused across domains and workflows. This article outlines how to model data to support natural language queries, accurately ground answers in authoritative sources, and enable [Atlas AI agents](/cdf/atlas_ai) to be reused across domains and workflows. AI agents use generative AI, specifically large language models (LLMs), to solve industrial business problems, like providing insights into historical and planned maintenance and analysis of time series data for root cause analysis. The agents include prompts with instructions, industry-relevant tools, and access to industrial data stored in your CDF knowledge graph. The agents have context about the underlying data model in CDF, including the [documentation you add to the data model definition](#documenting-the-data-model). ## Data model design When designing a data model for AI agents, consider the following: * **User-centric design**: Focus on the needs and expectations of the end-users who will interact with the AI agents. Understand their workflows and the types of queries they are likely to make. Would they, provided with the same context as the LLM, be able to answer the questions? * **Clarity and simplicity**: Strive for a clear and simple data model that is easy to understand and use. Avoid ambiguity. * **Consistency**: Maintain consistency in naming conventions, data types, and structures throughout the data model. This helps to reduce confusion and makes the model easier to work with. ### Data model concepts Data model concepts are the building blocks of the data model and define the structure and relationships of the data. For example, Atlas AI agents are often used with these core concepts and properties: * [**Assets**](/cdf/dm/dm_reference/dm_core_data_model#asset) — model the physical structure of the plant or facility, for example as an [asset hierarchy](/cdf/dm/dm_guides/dm_cdm_build_asset_hierarchy). The assets serve as contextual anchors for other concepts like time series, files, and activities. * [**Time series**](/cdf/dm/dm_reference/dm_core_data_model#timeseries) — link time series to assets and use clear and consistent [units](/cdf/dm/dm_reference/dm_core_data_model#unit). Time series lets you trend operational behavior and ground questions about status. * [**Files**](/cdf/dm/dm_reference/dm_core_data_model#file) — link documents to assets or equipment. Files lets you ask contextual questions or summarize information. * [**Activities**](/cdf/dm/dm_reference/dm_core_data_model#activity) — make sure activities like events and maintenance orders are properly typed and scoped. Activities allow agents to explain or correlate causes. * [**Relations**](/cdf/dm/dm_concepts/dm_spaces_instances#direct-relations-vs-edges) — define consistent relationships between concepts, like asset-to-file and activity-to-equipment. The relationships enable multi-hop queries and semantic linking. #### Example query To see how this works in practice, consider a prompt like: "*Is pump P-101 vibrating abnormally?*". To find and summarize the recommended intervention, an AI agent must: * Query the knowledge graph to identify P-101. * Search for files linked to P-101 (manuals, data sheets etc.). * Determine vibration abnormality thresholds by searching the files. * Query the linked time series for vibration data. * Analyze if the time series has been outside the thresholds. ## Documenting the data model To ensure that the data model is understandable and usable by AI agents, follow these best practices to document it. ### Use human-readable names Ideally, use human-readable names for [types and properties](/cdf/dm/dm_concepts/dm_property_graph#properties) and avoid abbreviations. Clear and concise property names enhance the understanding of the data model. ### Include short descriptions for types Add a brief description above each [type](/cdf/dm/dm_concepts/dm_property_graph#properties) to give context. For instance, if the type `Event` is used for work orders and alerts, specify this in the type's docstring. This helps the LLM understand that queries about work orders and alerts refer to `Event`. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} """ An abstraction of an event that occurs at some point in time. An Event can be a work order or an alert. For example: Open Valve A, Place scaffolding """ type Event {..} ``` ### Document every property Document every property within a [type](/cdf/dm/dm_concepts/dm_property_graph#properties). Include a clear description and provide an example value. Follow this convention: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ... type Operation { ... """ ID from the source system Example: 21003104 """ id: String ... } ``` ### Specify enum values If a [property](/cdf/dm/dm_concepts/dm_property_graph#properties) can only have specific values (is an `Enum`), create an enum type for the property. For example: ```graphql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} enum Status { Open Closed Released } type WorkOrder { """ Active status of the work order. """ status: Status } ``` ## Document parsing and confidence scores When you use [document parsing](/cdf/integration/guides/contextualization/document_parser) to extract data from documents into your data model, each extracted property gets a **confidence score** based on how well the data model property names match the field names in the document. * Use human-readable property names that match or closely resemble the field names in your documents (for example, datasheets and PDFs). * Avoid abbreviations and property names that differ markedly from those field names. Close alignment improves confidence scores and reduces corrections after parsing. * Design the view on a representative document so property names align with the field names before you run batch parsing. # Building an asset hierarchy Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_cdm_build_asset_hierarchy Learn how to build an asset hierarchy with the Cognite core data model and populate it using transformations. The [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) concept in the Cognite core data model has properties to represent systems that support industrial functions or processes. You can structure assets hierarchically according to your organization's criteria. For example, an asset can represent a **water pump** that's part of a larger **pump station** asset at a **clarification plant** asset. [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) comes with a matching [**container**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) and [**view**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views). The container has efficient [indexes](/cdf/dm/dm_concepts/dm_containers_views_datamodels#indexes) and [constraints](/cdf/dm/dm_concepts/dm_containers_views_datamodels#constraints) and can be used at scale out-of-the-box. ## Build an asset hierarchy with CogniteAsset We recommend building asset hierarchies starting from a **root** node and using the `parent` property to add **child** assets. Root nodes are nodes with an empty `parent` property. You can add or change the `parent` property later to add a node and its children to an existing hierarchy. When you add or change a node in a hierarchy, a background job—*path materializer*—recalculates the path from the root node for all nodes affected by the change. The job runs asynchronously, and the time it takes to update all the paths depends on the number of affected nodes. The example below uses [sample data](https://github.com/cognitedata/toolkit/blob/f9c2ce8c4de8928718acc48eb6688338c6b3508f/cognite_toolkit/cognite_modules/examples/example_pump_asset_hierarchy/raw/collections_pump.csv) from the [Cognite Data Fusion Toolkit](/cdf/deploy/cdf_toolkit). The hierarchy consists of two levels—**lift stations** and **pumps**—where **pumps** are children of a **lift station**. To create a generic asset hierarchy with the [**CogniteAsset view**](/cdf/dm/dm_reference/dm_core_data_model#asset), set the **CogniteAsset view** as a `source` during the ingestion process and set the `parent` property to specify the parent asset or leave it blank for root assets. ### Populate data with Transformations This example uses [Transformations](/cdf/integration/guides/transformation/transformations) to ingest data from the `lift-stations-pumps` table from the [CDF staging area](/cdf/integration/guides/extraction/raw_explorer), RAW. Upload the data to the staging area, for example, using a script similar to this: You can find examples and documentation for [Cognite Python SDK](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/index.html) here: * [Authentication](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/credential_providers.html#) * [RAW](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#raw) ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import pandas as pd from cognite.client import ClientConfig, CogniteClient # client_config = ... set up you authentication here client = CogniteClient(client_config) # create RAW database and table raw_database = client.raw.databases.create("cdm-asset-demo") raw_table = client.raw.tables.create(raw_database.name, "lift-stations-pumps") # use pandas to read csv file pump_data_url = "https://github.com/cognitedata/toolkit/raw/f9c2ce8c4de8928718acc48eb6688338c6b3508f/cognite_toolkit/cognite_modules/examples/example_pump_asset_hierarchy/raw/collections_pump.csv" lift_station_pumps_frame = pd.read_csv(pump_data_url) # upload dataframe to RAW client.raw.rows.insert_dataframe(raw_database.name, raw_table.name, lift_station_pumps_frame) ``` Create a [space](/cdf/dm/dm_concepts/dm_spaces_instances#space) named **`asset-demo`** to hold asset instances. This is separate from the RAW database name (`cdm-asset-demo`). Create the space using one of these approaches: * **Programmatically** — Use [Apply spaces](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/generated/cognite.client.AsyncCogniteClient.data_modeling.spaces.apply.html) in the Cognite Python SDK reference. * **In the transformation UI** — In the next step, select + Create new space in the **Target space** dropdown and enter **`asset-demo`**. 1. Navigate to Data fusion > Integrate > Transformations. 2. Select **Create transformation**, and enter a unique **name** and a unique **external ID**. 3. Select **Next**. 4. Select **Cognite core data model** under **Target data models**, then select **v1** for **Version**. 5. For **Target space**, select **`asset-demo`**. If you created the space programmatically in the previous step, select it from the dropdown. Otherwise, select + Create new space and enter **`asset-demo`**. 6. Select **CogniteAsset** as the **Target type or relationship**. 7. Select **Create or Update** as the **Action**. 8. Select **Create**. asset transformation Add and **run** this SQL specification to create the asset hierarchy: ```sql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} -- Add the root manually. The set for lift stations doesn't specify a root asset. select 'lift_pump_stations:root' as externalId, 'Lift Station Root' as name, 'Root node to group all lift stations.' as description, null as parent UNION ALL -- select lift station data select distinct concat('lift_station:', cast(`LiftStationID` as STRING)) as externalId, cast(`LiftStationID` as STRING) as name, cast(`LocationDescription` as STRING) as description, node_reference('asset-demo','lift_pump_stations:root') as parent from `cdm-asset-demo`.`lift-stations-pumps` where isnotnull(nullif(cast(`LiftStationID` as STRING), ' ')) UNION ALL -- select pump data select concat('pump:', cast(`FacilityID` as STRING)) as externalId, cast(`Position` as STRING) as name, null as description, node_reference('asset-demo', concat('lift_station:', cast(`LiftStationID` as STRING))) as parent from `cdm-asset-demo`.`lift-stations-pumps` where isnotnull(nullif(cast(`LiftStationID` as STRING), ' ')); ``` ## Build an asset hierarchy and extend CogniteAsset Using the generic **CogniteAsset** in combination with other **core data model concepts** is often enough to create the necessary data models. If you need to create custom data models, you can **extend** the Cognite data models and tailor them to your use case, solution, or application. The example below builds on the [previous section](#build-an-asset-hierarchy-with-cogniteasset). Complete the RAW upload and create the **`asset-demo`** space before defining custom schema components. The example shows how to extend **CogniteAsset** to hold more pump data: `DesignPointFlowGPM`, `DesignPointHeadFT`, `LowHeadFT`, and `LowHeadFlowGPM`. The first step is to create a [container](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) to hold the additional data: ```yaml title="Pump container definition" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: Pump name: Pump space: asset-demo usedFor: node properties: DesignPointFlowGPM: autoIncrement: false defaultValue: null description: The flow the pump was designed for, specified in gallons per minute. name: DesignPointFlowGPM nullable: true type: list: false type: float64 DesignPointHeadFT: autoIncrement: false defaultValue: null description: The flow head pump was designed for, specified in feet. name: DesignPointHeadFT nullable: true type: list: false type: float64 LowHeadFT: autoIncrement: false defaultValue: null description: The low head of the pump, specified in feet. name: LowHeadFT nullable: true type: list: false type: float64 LowHeadFlowGPM: autoIncrement: false defaultValue: null description: The low head flow of the pump, specified in gallons per minute. name: LowHeadFlowGPM nullable: true type: list: false type: float64 constraints: requiredAsset: constraintType: requires require: type: container space: cdf_cdm externalId: CogniteAsset ``` The `requires` [constraint](/cdf/dm/dm_concepts/dm_containers_views_datamodels#constraints) ensures that the **Pump** data is always accompanied with **CogniteAsset** data. Next, create two views to present the nodes as **lift-station** and **pump** assets. By [implementing](/cdf/dm/dm_concepts/dm_containers_views_datamodels#implementing-other-views) the **CogniteAsset** view, you automatically include all the properties defined in that view: ```yaml {5-8} title="Pump view definition" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: Pump name: Pump space: asset-demo version: v1 implements: # <-- Declares that the view implements the CogniteAsset view - space: cdf_cdm externalId: CogniteAsset version: v1 properties: DesignPointFlowGPM: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: DesignPointFlowGPM name: DesignPointFlowGPM DesignPointHeadFT: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: DesignPointHeadFT name: DesignPointHeadFT LowHeadFT: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: LowHeadFT name: LowHeadFT LowHeadFlowGPM: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: LowHeadFlowGPM name: LowHeadFlowGPM ``` ```yaml {4-7} title="LiftStation view definition" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} space: asset-demo externalId: LiftStation version: v1 implements: # <-- Declares that the view implements the CogniteAsset view - space: cdf_cdm externalId: CogniteAsset version: v1 filter: # <-- Adds additional filter to only include lift-stations prefix: property: - node - externalId value: lift_station ``` Note the filter property in the **LiftStation** view. The **Pump** view includes properties from the **Pump** container and the **core data model** containers. The default [`hasData`](/cdf/dm/dm_concepts/dm_querying#hasdata-filter) filter ensures that the view only presents instances that are **pumps** or have **pump** data. The **LiftStation** view has the same fields as the **CogniteAsset** view. But without the explicit filter, it returns both **pumps** and **lift stations**. When using explicit filters in views, it's essential to consider query performance. View filters run every time someone uses the view to query data. If necessary, add [indexes](/cdf/dm/dm_concepts/dm_containers_views_datamodels#indexes) to avoid degraded performance. The system always indexes the `externalId` property. View filters don't limit any write operations based on the filter expression. You can ingest an instance through a view without it being returned by the same view. When ingesting instance data through a view as a `source`, the system ensures that all the required fields are populated. When querying, using the `hasData` filter for a view ensures that the returned results have all the required fields for the view. Even if a view has no required fields, you can use the view to write "*blank*" values to satisfy the conditions for the `hasData` filter. To populate the custom asset hierarchy, create a [data model](/cdf/dm/dm_concepts/dm_containers_views_datamodels#data-models) to present the custom views: ```yaml title="Lift stations and pumps data model" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: liftStationsPumps name: Lift stations and pumps space: asset-demo version: v1 views: - externalId: LiftStation space: asset-demo type: view version: v1 - externalId: Pump space: asset-demo type: view version: v1 ``` The below Python script shows how to use the Cognite Python SDK and the `.load()` function to create data modeling objects from **YAML** representation. Remember to [authenticate](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/credential_providers.html#) the **`CogniteClient`** properly. ```python title="Creating custom schema components" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import ClientConfig, CogniteClient from cognite.client.data_classes.data_modeling import ( ContainerApply, DataModelApply, ViewApply, ) # client_config = ... <-- remember to authenticate. client = CogniteClient(client_config) pump_container_yaml = """ externalId: Pump name: Pump space: asset-demo usedFor: node properties: DesignPointFlowGPM: autoIncrement: false defaultValue: null description: The flow the pump was designed for, specified in gallons per minute. name: DesignPointFlowGPM nullable: true type: list: false type: float64 DesignPointHeadFT: autoIncrement: false defaultValue: null description: The flow head pump was designed for, specified in feet. name: DesignPointHeadFT nullable: true type: list: false type: float64 LowHeadFT: autoIncrement: false defaultValue: null description: The low head of the pump, specified in feet. name: LowHeadFT nullable: true type: list: false type: float64 LowHeadFlowGPM: autoIncrement: false defaultValue: null description: The low head flow of the pump, specified in gallons per minute. name: LowHeadFlowGPM nullable: true type: list: false type: float64 constraints: requiredAsset: constraintType: requires require: type: container space: cdf_cdm externalId: CogniteAsset """ pump_view_yaml = """ externalId: Pump name: Pump space: asset-demo version: v1 implements: # <-- Declares that the view implements the CogniteAsset view - space: cdf_cdm externalId: CogniteAsset version: v1 properties: DesignPointFlowGPM: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: DesignPointFlowGPM name: DesignPointFlowGPM DesignPointHeadFT: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: DesignPointHeadFT name: DesignPointHeadFT LowHeadFT: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: LowHeadFT name: LowHeadFT LowHeadFlowGPM: container: externalId: Pump space: asset-demo type: container containerPropertyIdentifier: LowHeadFlowGPM name: LowHeadFlowGPM """ lift_station_view_yaml = """ space: asset-demo externalId: LiftStation version: v1 implements: # <-- Declares that the view implements the CogniteAsset view - space: cdf_cdm externalId: CogniteAsset version: v1 filter: # <-- Adds additional filter to only include lift-stations prefix: property: - node - externalId value: lift_station """ data_model_yaml = """ externalId: liftStationsPumps name: Lift stations and pumps space: asset-demo version: v1 views: - externalId: LiftStation space: asset-demo type: view version: v1 - externalId: Pump space: asset-demo type: view version: v1 """ pump_container = ContainerApply.load(pump_container_yaml) client.data_modeling.containers.apply(pump_container) pump_view = ViewApply.load(pump_view_yaml) lift_station_view = ViewApply.load(lift_station_view_yaml) client.data_modeling.views.apply([pump_view, lift_station_view]) data_model = DataModelApply.load(data_model_yaml) client.data_modeling.data_models.apply(data_model) ``` In the [previous example](#build-an-asset-hierarchy-with-cogniteasset), we could use the **CogniteAsset** view to write data for **lift stations** and **pumps**. Because **Transformations** support writing to a **single** view, you need to create a transformation for each of the **LiftStation** and **Pump** views to write to **both**. Follow the steps in the [previous example](#build-an-asset-hierarchy-with-cogniteasset) to create transformations for the **LiftStation** and **Pump** views. Select **Lift stations and pumps** as the target data model. asset transformation Add and run these SQL specifications: ```SQL title="Transformation for Pumps" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} select concat('pump:', cast(`FacilityID` as STRING)) as externalId, cast(`Position` as STRING) as name, concat(cast(`Position` as STRING), " " , cast(`FacilityID` as STRING)) as description, node_reference('asset-demo', concat('lift_station:', cast(`LiftStationID` as STRING))) as parent, double(`DesignPointHeadFT`) as DesignPointHeadFT, double(`DesignPointFlowGPM`) as DesignPointFlowGPM, double(`LowHeadFT`) as LowHeadFT, double(`LowHeadFlowGPM`) as LowHeadFlowGPM from `cdm-asset-demo`.`lift-stations-pumps` where isnotnull(nullif(cast(`LiftStationID` as STRING), ' ')); ``` ```SQL title="Transformation for LiftStations" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} -- Add root manual root node as the set does not have a root for lift stations select 'lift_pump_stations:root' as externalId, 'Lift Station Root' as name, 'Root node to group all lift stations.' as description, null as parent UNION ALL -- select lift station data select distinct concat('lift_station:', cast(`LiftStationID` as STRING)) as externalId, cast(`LiftStationID` as STRING) as name, cast(`LocationDescription` as STRING) as description, node_reference('asset-demo','lift_pump_stations:root') as parent from `cdm-asset-demo`.`lift-stations-pumps` where isnotnull(nullif(cast(`LiftStationID` as STRING), ' ')) ``` ## Query hierarchical data The examples below show how to query hierarchical data using [**Cognite Python SDK**](https://cognite-sdk-python.readthedocs-hosted.com/en/latest). ### List by source view Views are a specialized representation of data and excellent tools for querying and filtering. For example, you can use the **CogniteAsset**, **LiftStation**, and **Pump** views to fetch nodes: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} assets = client.data_modeling.instances.list(sources=ViewId(space="cdf_cdm", external_id="CogniteAsset", version="v1"), space="asset-demo") lift_stations = client.data_modeling.instances.list(sources=ViewId(space="asset-demo", external_id="LiftStation", version="v1")) pumps = client.data_modeling.instances.list(sources=ViewId(space="asset-demo", external_id="Pump", version="v1")) ``` Pass `space` when using **CogniteAsset** so you only get assets from the `asset-demo` space, not other spaces in your project. Setting a view as a source when listing instances automatically applies a `hasData` filter to the query. When fetching instances with **CogniteAsset** as a source, all matching assets in the space are returned, both **lift-stations** and **pumps**. The respective views implement the **CogniteAsset** view and therefore satisfy the `hasData` filter. To inspect the results, iterate over the returned list. See [List instances](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/generated/cognite.client.AsyncCogniteClient.data_modeling.instances.list.html) in the Cognite Python SDK reference. ### Aggregate based on a common parent Often, you want to find aggregated information about the children of assets. This example finds information on how many **pumps** there are in a **lift station** and the average value of the `LowHeadFT` property for all **pumps** in a **lift station**: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes import aggregations from cognite.client.data_classes.filters import Equals # count number of pumps for Riverhouse lift station count_pumps_for_lift_station = client.data_modeling.instances.aggregate( view=ViewId(space="asset-demo", external_id="Pump", version="v1"), space="asset-demo", filter=Equals( property=["asset-demo", "Pump/v1", "parent"], value={"space": "asset-demo", "externalId": "lift_station:Riverhouse"}, ), aggregates=aggregations.Count("externalId"), ).value # average pumpOn value for lift-station someId average_for_for_lift_station = client.data_modeling.instances.aggregate( view=ViewId(space="asset-demo", external_id="Pump", version="v1"), space="asset-demo", filter=Equals( property=["asset-demo", "Pump/v1", "parent"], value={"space": "asset-demo", "externalId": "lift_station:Riverhouse"}, ), aggregates=aggregations.Avg("LowHeadFT"), ).value ``` ### List all assets in a subtree To list all descendants of an asset, use the system-maintained `path` property of the **CogniteAsset** view. Use `path` for subtree traversal and filtering. The `children` property in **CogniteAsset** is a [reverse direct relation](/cdf/dm/dm_concepts/dm_containers_views_datamodels#reverse-direct-relation) resolved at query time and has different query behavior than `root`, `path`, and `pathLastUpdatedTime`. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.data_modeling import NodeId from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteAsset from cognite.client.data_classes.filters import Prefix sub_tree_root = client.data_modeling.instances.retrieve_nodes( NodeId("asset-demo", "lift_station:Riverhouse"), node_cls=CogniteAsset ) sub_tree_nodes = client.data_modeling.instances.list( instance_type=CogniteAsset, filter=Prefix(property=["cdf_cdm", "CogniteAsset/v1", "path"], value=sub_tree_root.path), ) ``` The example uses **CogniteAsset** **TypedNode** from the core data model definitions, a helper class for *de-serialization* of data objects. # Delete nodes without associated container data Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_clean_up_nodes_without_data Use a Python script to delete nodes without associated container data in your data model. We strongly recommend first running the script in `dry-run` mode to determine which nodes it'll delete. Make sure that the user or service account running the script has the `datamodels:read:all` capability. The script returns only the containers it has access to. If the script has limited container access, it may delete instances that still have data. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.capabilities import DataModelsAcl from cognite.client.data_classes.data_modeling.instances import InvolvedContainers from cognite.client.data_classes.filters import MatchAll, SpaceFilter from cognite.client import CogniteClient client = CogniteClient() # Make sure that the principal running the script has the `datamodels:read:all`capability. # The script returns only the containers it has access to. If the script has limited container access, # it may delete instances that still have data. to_check = [ DataModelsAcl( actions=[DataModelsAcl.Action.Read], scope=DataModelsAcl.Scope.All()) ] if missing := client.iam.verify_capabilities(to_check): raise Exception(f"You don't have the necessary capabilities to scan all containers: {missing}") # Set dry_run to False to delete the nodes. Set to True to list the nodes without deleting them. dry_run = True print(f"Dry run: {dry_run}") if dry_run: print("This is a dry run. The nodes will be listed, but not deleted.") else: print("This is NOT a dry run. The nodes will be deleted.") # Replace this filter to scope which nodes to delete filter = MatchAll() # You can, for example, use a space filter: # filter = SpaceFilter("myspace") # Type nodes must be deleted _after_ you have deleted the instances pointing to them. # You can use a filter to exclude the space where you store the type nodes. # filter = Not(SpaceFilter("my_type_node_space")) # If you haven't stored your type nodes in a dedicated space and need to filter them out by # external ID, do that on the client side to prevent the query performance from degrading. deleted_total = 0 for batch in client.data_modeling.instances(instance_type="node", filter=filter, chunk_size=1000): result = client.data_modeling.instances.inspect( nodes=batch.as_ids(), involved_containers=InvolvedContainers() ) nodes_without_container_data = [ (node.space, node.external_id) for node in result.nodes if not node.inspection_results.involved_containers ] if dry_run: for space, external_id in nodes_without_container_data: print(f"{space}:{external_id}") else: client.data_modeling.instances.delete(nodes=nodes_without_container_data) deleted_total += len(nodes_without_container_data) print(f"Deleted {len(nodes_without_container_data)} nodes without container data") print(f"Deleted a total of {deleted_total} nodes") ``` # Debugging query performance Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_debug_query_performance Learn how to use debug notices to understand query execution and identify optimization opportunities for your data model queries. Query performance directly impacts your application's scalability. **Debug notices** help you understand how your queries run internally and identify opportunities for optimization. See the [Performance considerations](/cdf/dm/dm_guides/dm_performance_considerations) article for core principles and best practices for optimizing queries. Use debug notices to see details about filtering efficiency, index usage, sorting behavior, and execution plans. Debug notices highlight common performance patterns but **don't detect all inefficiencies**. For full details on debug notices, see the alpha version of the API specification. Look for the schema under `datamodels/instances/query` or `datamodels/instances/sync` sections. The request schema shows configuration options, and the response schema lists all possible debug notices. ## Enable debug notices To enable debug notices in your queries: Add a `debug` object to your API request to activate debug notices: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "debug": {} } ``` If your query times out, increase the timeout value and disable query results output: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "debug": { "emitResults": false, "timeout": 20000 } } ``` The maximum timeout is 60,000 milliseconds (60 seconds). Specify timeout values in milliseconds. ## Understand grades and profiling Debug notices highlight both **positive** and **negative** aspects of your query execution. They warn about potential performance issues and confirm when your query is well optimized. ### Grades Each debug notice includes a **grade** that shows the importance or impact of the notice. Use this grading system to prioritize issues and recognize good practices: | Grade | Description | | - | - | | A | Excellent; best practice followed | | B | Good; minor optimizations possible | | C | Moderate concern; worth reviewing | | D | Significant issue; likely to affect performance | | E | Critical issue; urgent attention needed | The grading is conservative by default. Enable detailed profiling by setting `profile: true` to see the full spectrum, including critical warnings (**E**) and strong positive confirmations (**A**). Profiling provides deeper insights for more accurate grading. ### Enable profiling The `profile` option enables detailed query profiling, which gathers runtime statistics like execution timing and resource usage for each query part. Profiling focuses on internal diagnostics, and you must disable query results by setting `emitResults` to `false`. Use this analysis for complex queries to uncover hidden performance bottlenecks. To activate profiling: Activate profiling with this basic example: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "debug": { "emitResults": false, "profile": true, "timeout": 30000 } } ``` ## Request execution plans **Alpha:** The includePlan feature is in alpha. The format of the execution plan in the response is not guaranteed to be stable and may change in future versions. Do not build production systems that rely on the specific structure of this output. Request the underlying PostgreSQL execution plan as part of debug notices by enabling the appropriate debug options. The execution plans show how the query processes at the database level. Execution plans help you analyze and diagnose query performance issues. If you have advanced PostgreSQL knowledge, you can also use these plans to understand query execution strategy and identify optimizations. To include execution plans: Include execution plans with this minimal example: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "debug": { "includePlan": true } } ``` ## Request translated queries **Alpha:** The translatedQuery feature is in alpha. The format of the translated query in the response is not guaranteed to be stable and may change in future versions. Do not build production systems that rely on the specific structure of this output. Request the underlying query representation by enabling the translatedQuery debug option. This feature is intended for advanced debugging. To include the translated query: Include the translated query with this minimal example: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "debug": { "translatedQuery": true } } ``` ## Handle common debug notices This section focuses on specific debug notices the API returns and how to resolve them. ### Quick reference | Debug notice | Issue type | Quick fix | | - | - | - | | [`sortNotBackedByIndex`](#sortnotbackedbyindex) | Missing index | Add matching sort index. | | [`unindexedThrough`](#unindexedthrough) | Missing index | Add index on relation target. | | [`significantPostFiltering`](#significantpostfiltering) | Query structure | Move filters earlier. | | [`significantHasDataFiltering`](#significanthasdatafiltering) | View complexity | Add requires constraints. | ### `sortNotBackedByIndex` **Problem:** Your query uses a sort order without a *cursorable* index. The system performs an in-memory sort at query time, which increases resource usage and slows performance as data scales. **Solution:** Ensure your data model includes a cursorable index that matches your query's sort order. This matters especially when sorting by multiple properties. Your query's sort order must exactly match the property order in the index for the database to use it efficiently. To resolve this issue: If your query contains this sort: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sort": [ { "property": ["space", "container", "A"], "direction": "ascending" }, { "property": ["space", "container", "B"], "direction": "descending" } ] } ``` Ensure that `"container"` has a compound index on properties `A,B` in that order. The order matters, and an index on `B,A` won't work. Create the index using [NEAT](/cdf/deploy/neat/validation), Schema Service, or the DMS API. To create the index using the DMS API, send this request to `models/containers`: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "space": "space", "externalId": "container", "indexes": { "index-identifier": { "properties": ["A", "B"], "cursorable": true } } } ] } ``` Replace `index-identifier` with an appropriate identifier for your index. ### `unindexedThrough` **Problem:** Your query uses a `through` clause to traverse a direct relation. This notice appears when the traversal direction requires a missing index. Traversing a direct relation `inwards` requires an index on the relation's target. Without it, the query scans many nodes to find the ones pointing to the source, which is inefficient and won't scale. **Solution:** Define an index on the property that is the target of the inwards direct relation. To resolve this issue: If your query looks like this: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "with": { "0": { "nodes": { "filter": { "hasData": [ [ "space", "viewA", "1" ] ] } } }, "1": { "edges": { "from": "0", "through": { "source": { "type": "container", "space": "space", "externalId": "containerB" }, "identifier": "directRelationB", "direction": "inwards" } } }, "select": { "1": {} } } } ``` Because the direction is "inwards", the query engine needs to efficiently look up nodes in "containerB" that have a direct relation named "directRelationB" pointing to the nodes from step "0". Create an index on the "directRelationB" property within "containerB" as shown in the [`sortNotBackedByIndex`](#sortnotbackedbyindex) section. ### `significantPostFiltering` **Problem:** The query engine processed many intermediate rows before filtering them to the final result set. This is inefficient because significant work goes to data that gets discarded. This happens when filters aren't selective enough in early stages of a chained query. **Solution:** Review your query to introduce more selective filters earlier. Use your knowledge about the data graph to narrow intermediate results as much as possible. To resolve this issue: Consider this query that fetches all pumps in space "SiteA" with an edge from another pump: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "with": { "0": { "nodes": { "filter": { "hasData": [["schema", "Pump", "v1"]] }}}, "1": { "edges": { "from": "0" }}, "2": { "nodes": { "from": "1", "filter": { "and": [ { "hasData": [["schema", "Pump", "v1"]] }, { "equals": {"property": ["node", "space"], "value": "SiteA"} } ] } } } }, "select": { "2": {} } } ``` Steps 0 and 1 could fetch many pumps and connecting edges from all sites before step 2 filters them to just "SiteA". Use additional graph knowledge to make the query more efficient. If you know all relevant pumps are at the same site, move the space filter to the first step. If you know the relevant edges are always type "FLOWS\_TO", add this filter to step 1. Apply these filters early to dramatically reduce the data processed in subsequent steps for better performance. **Modified and improved query:** ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "with": { "0": { "nodes": { "filter": { "and": [ { "hasData": [["schema", "Pump", "v1"]] }, { "equals": { "property": ["node", "space"], "value": "SiteA" } } ] } } }, "1": { "edges": { "from": "0", "filter": { "equals": {"property": ["edge", "type"], "value": {"space": "types", "externalId": "FLOWS_TO"}} } } }, "2": { "nodes": { "from": "1", "filter": { "hasData": [["schema", "Pump", "v1"]] } } } }, "select": { "2": {} } } ``` ### `significantHasDataFiltering` **Problem:** A `hasData` filter applies to a view composed of many underlying containers. To satisfy the filter, the query engine may perform a join operation for each container to check for data existence. This becomes computationally expensive and slows your query. **Solution:** Help the query planner optimize your query by adding `requires` constraints to container definitions used in the query. If you know your data model, for example, that data presence in one container implies data presence in another, use this to help the query engine prune unnecessary joins. To resolve this issue: Suppose you have view `V` that maps three containers: `A`, `B`, and `C`. You know that any node with data in container `B` or `C` is guaranteed to have data in container `A`. Without optimization, a `hasData` filter on `V` checks for data in `A` AND `B` AND `C`, resulting in multiple joins. By defining requires constraints on containers `B` and `C` that point to `A`, you inform the query engine of this dependency. When it sees a `hasData` filter on `V`, it understands it only needs to check for data in container `A`. This simplifies the query, reduces joins, and can significantly improve performance. # Designing scalable data models Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_designing_scalable_models Learn best practices for designing scalable data models in CDF, including layered architecture, design principles, performance optimization, security, and deployment. This guide outlines a strategic approach to designing scalable and efficient data models in Cognite Data Fusion (CDF). It covers layered architecture (source, enterprise, solution), containers, views, design principles, and deployment practices for maintainability, scalability, and security. **Prerequisites:** Familiarity with the [data modeling service](/cdf/dm/) and core CDF concepts ([containers, views, and data models](/cdf/dm/dm_concepts/dm_containers_views_datamodels), [spaces and instances](/cdf/dm/dm_concepts/dm_spaces_instances)) will help you get the most from this guide. ## Why it matters A well-designed data model reduces integration complexity, improves query performance, and makes it easier to scale as your organization grows. Poor design leads to slow queries, maintenance overhead, and difficulty adding new data sources or applications. Following the principles in this guide helps you avoid common pitfalls and build models that support both operational and analytical use cases. ## Layered architecture A layered data model architecture separates responsibilities and optimizes for different use cases. Clear ownership at each layer improves maintainability: **Data producers** focus on quality and freshness, **data modelers** on integration and business logic, and **data consumers** on application-specific views. This structure also scales better as your organization grows, because changes in one layer do not cascade unpredictably to others. The three layers map loosely to the medallion architecture (bronze, silver, gold). See [medallion architecture](https://dataengineering.wiki/Concepts/Data+Architecture/Medallion+Architecture) for an overview. Medallion architecture diagram showing bronze, silver, and gold layers | Layer | Owner | Purpose | Medallion analogy | | :- | :- | :- | :- | | **Source** | Data producers | Raw copy of data from a single source system. Ensures availability and quality. Responsibilities include building ELT pipelines into CDF, managing upstream lineage, and monitoring source data. | Silver | | **Enterprise** | Data modelers | Integrates and contextualizes data from multiple sources. Serves as the single, validated source of truth. | Gold | | **Solution** | Data consumers | Tailored, read-only views for specific applications. Data consumers specify requirements and work with the data modeler to find data in the enterprise data model, helping define new solution schemas as needed. | Application-specific | As a general rule, map data into solution [views](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views) directly from enterprise data model [containers](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) rather than transforming or duplicating the data. In specific cases, a solution model can include its own containers (populated by discrete workflows) to store data unique to a particular use case or application. **Write-back principle:** For write-back use cases, data must be written to a **separate source data model** (owned by a data producer), never directly into the enterprise layer. You can use a simplified layered architecture to get started faster (for example, with only two layers). The full three-layer architecture described in this section is the recommended approach for enterprise-scale deployments. ## Core design principles Beyond layered architecture, a few design principles keep your data models efficient and scalable. These apply whether you are building a source, enterprise, or solution model. ### Avoid duplication via shared IDs Represent each entity only once in the knowledge graph. You link data across views and layers by reusing the **same instance ID** ([space](/cdf/dm/dm_concepts/dm_spaces_instances#space) + external ID). [Direct relations](/cdf/dm/dm_concepts/dm_spaces_instances#direct-relations-vs-edges) are usually only required between different entities and not between different dimensions of the same entity. ### Extend core data models (CDM) When you build your enterprise data model, prefer **custom views that implement** the relevant base [core data model (CDM)](/cdf/dm/dm_reference/dm_core_data_model) views instead of using the CDM views directly. CDM is a blueprint for common properties, but you will usually add domain-specific properties and relationships for your **use cases**. Because you cannot change CDM views or their relations, linking custom views to them can break the schema or create **dead ends** where you cannot walk the graph back to your custom entity. Usually, extend the `CogniteAsset`, `CogniteEquipment`, `CogniteFile`, `CogniteTimeSeries`, and `CogniteActivity` views when you plan to populate them with data. Skip extending any of these that you do not need. Other CDM views are often used directly by CDF applications, for example `CogniteFileCategory` and the 3D views `Cognite3DModel`, `Cognite3DModelRevision`, and `Cognite3DModelRevisionFile`. You typically do not need to extend those unless a custom application reads from them. Asset type implementing CogniteAsset with relations to LifeCycleInformation and MaintenanceAndIntegrity data domains, and TimeseriesReference back to the central Asset type Balance the number of views. The more views you add, the more transformations and resources are needed to populate the data. ### Distribute properties contextually Distribute properties for the same instance across multiple context-specific containers and views. Containers are physical 'property groups', and therefore are a great way to group context relevant properties of an instance together, while views can consist of a combination of these. As a general rule of thumb, you should try to avoid both very wide and very narrow containers and views, in order to ensure optimal performance and to avoid schema proliferation. Following this principle also improves AI accuracy by scoping containers and views to a relevant set of properties for a particular context and use case. For example, omit pipe-coating properties from a view focused on pump equipment. Finally, both containers and views have [**property count limits**](/cdf/dm/dm_reference/dm_limits_and_restrictions), which are set on a CDF project-level, and this practice can also help you adhere to these limits. ### Model relationships Choose the right relationship type for your use case. [Direct relations](/cdf/dm/dm_concepts/dm_spaces_instances#direct-relations-vs-edges) are the default. They are simpler, faster, and less resource-intensive than edges. Add a reverse direct relation when you need to navigate the graph in both directions. Use [edges](/cdf/dm/dm_concepts/dm_spaces_instances#direct-relations-vs-edges) when you need metadata on relationships, or when you have dynamic and potentially very high cardinality on a many-to-many relationsip. Edges have unbounded cardinality, while a single relationship represented by a list of direct relations currently only support up to [2,000 outgoing direct relations](/cdf/dm/dm_reference/dm_limits_and_restrictions#list-size-limits) from one instance. Edges add complexity and cost, they count as instances and hence use up instance quota limits. Additionally, supported query patterns for edges vary from direct relations, which needs to be considered in the design. | Type | When to use | Trade-off | | :- | :- | :- | | **Direct relations** | Default for most relationships. Include reverse direct relation. | Simpler, faster, less resource-intensive | | **Edges** | Relationship properties, exceed 2,000 linked instances, graph traversal with extra properties | More complex, higher cost | ## Query and performance optimization Indexing and constraints are essential for efficiency and to prevent timeouts. Without them, queries on large graphs can become unusably slow. Add **B-tree indexes** to scalar properties you filter on and to direct relations you use for graph traversal, including reverse direct relations. Use **inverted indexes** for list-type properties. You can define up to [**10 indexes per container**](/cdf/dm/dm_reference/dm_limits_and_restrictions). For more detail, read [Performance considerations](/cdf/dm/dm_guides/dm_performance_considerations). For any view that spans multiple containers, **requires** container constraints are mandatory: they let the query engine apply essential optimizations. Without them, performance is typically unacceptable. Use human-readable names and descriptions for all properties to aid users and AI agents (Atlas AI). Denormalize properties that are frequently queried and give additional context, since this helps AI agents, applications, and human users alike. See [Optimizing data models for AI](/cdf/dm/dm_guides/dm_best_practices_ai_search) for more guidance. ## Security and governance Separating schema from data instances enables granular [access control](/cdf/dm/dm_concepts/dm_access_control). Use distinct **schema spaces** (for model definitions) and **instance spaces** (for data), and rely on tooling for controlled deployment. The permission model distinguishes schema from data: `datamodels:read` gives schema access, while **`datamodelinstances:read`** is required to read any data. Time series and files are special cases. Conceptually, they are meta/container types: the data model instance (the node) points to the actual bulk storage for the instance's data (the datapoints or file bytes). To control access, you need two distinct types of permissions: 1. **Schema access:** You need **`datamodels:read`** on the **`cdf_cdm`** space. This is required for your application to understand the structure and definition of what a CogniteTimeSeries or CogniteFile is. 2. **Data access:** You need **`datamodelinstances:read`** on the **instance space** (for example, `my-asset-space`). This grants access to read the specific instances and their associated time series datapoints or file bytes. Using distinct instance spaces is the essential strategy for segregating data access. Separate instances by creating a space for each location (for example, `sp_location_a_instances`) to control data access using CDF groups. Consider limits on space count within the project when implementing a strategy that subdivides data by source system, location, or use case. Combining these could result in exponential growth in space usage for enterprise-scale deployments. ### Source and enterprise layer instance strategy Define a strategy for instances between the source and enterprise layers. In the solution layer, you generally map data within the enterprise layer. **Separation** prevents data deletion risk and enables data isolation (for example, for third-party suppliers who may be competitors) but increases the instance count. **Combining** reduces instances and simplifies mapping, but risks unintended data loss. Extra care is needed when making changes in the source layer, since deleting an instance in a source model would delete the same instance in the enterprise model. **Recommendation:** Use **separate instances for the source and enterprise layers**, where each domain owns the instance space it writes to. For example, a PI extractor would create a source model with an instance space owned by the source domain. The derived domain (enterprise model) uses a separate instance space owned by the enterprise layer. ## Deployment Govern and version your model in Git, and deploy it with configuration as code rather than manual changes in the UI. Two tools support this, depending on who is doing the modeling work: * **Data modelers**, working directly in YAML: use the [Cognite Toolkit](/cdf/deploy/cdf_toolkit) to build, validate, and deploy across Dev/Staging/Prod environments through CI/CD. * **Domain subject matter experts**, working in spreadsheets: use [NEAT](/cdf/deploy/neat) to design and validate the model, then convert it to YAML and deploy it with the Cognite Toolkit. NEAT does not replace the Cognite Toolkit for deployment. ## Naming and versioning Follow the [resource naming conventions](/cdf/deploy/reference/cdf_resource_naming_conventions) for external IDs and properties, and use [Cognite Toolkit variables](/cdf/deploy/cdf_toolkit/api/config_yaml#the-variables-section) to configure names and versions across environments. Bump the view version with every schema change. ## Validation checklist Before deploying or merging changes, run through this checklist to confirm your model follows the best practices in this guide. The following table lists verification items by category: layered architecture, core design, query and performance, governance, deployment, and naming. | Category | Checklist item | Check | | :- | :- | :-: | | Layered architecture | Source data model(s) exist (raw copy of data from a single source). | ☐ | | | Enterprise data model(s) exist (data aggregates from multiple sources). | ☐ | | | Enterprise data model(s) provide data that is generic and understandable without source system knowledge. | ☐ | | | Solution data model(s) exist (provided for specific use cases). | ☐ | | | Solution models write-back data to a separate source data model. | ☐ | | Core design and scalability | Properties are distributed across contextual containers and views. | ☐ | | | Custom views extend (implement) the necessary base CDM types. | ☐ | | | CogniteAsset is extended only once and linked to domain-specific dimensions. | ☐ | | | Unnecessary data duplication is avoided. | ☐ | | | Models use shared instance IDs to enable flexible linking. | ☐ | | | Direct relations are used as the default relationship. | ☐ | | Query and performance | B-tree and inverted indexes are used on frequently filtered properties. | ☐ | | | Requires container constraints are used on views that implement others. | ☐ | | | Human-readable names and descriptions are used for all properties to aid users and AI agents. | ☐ | | | Properties that are frequently queried and give additional context are denormalized. | ☐ | | Governance and security | Schema and instance spaces are kept separate. | ☐ | | | Access control is set up per facility, plant, or site. | ☐ | | | A clear instance strategy for the source and enterprise layers is defined. The recommendation is to separate. | ☐ | | Development and deployment | The Cognite Toolkit is used to manage resources. | ☐ | | | A source code management platform like Git is used. | ☐ | | | NEAT is considered as one option to develop data models, where customer subject matter experts (SMEs) are heavily involved. | ☐ | | Naming and versioning | An explicitly defined naming convention is in place, referencing the defined best practices. | ☐ | | | A numerical versioning pattern is used. | ☐ | | | View versions are incremented for every schema change. | ☐ | | | Cognite Toolkit variables are used to configure names (for example, space names) and versions. | ☐ | ## Tools and services The following services and tools support data modeling in CDF. Choose the right tool based on your role and workflow. | Service | Description | | :- | :- | | [Data modeling service](/cdf/dm/) | The engine within CDF that builds and manages an Industrial Knowledge Graph. Uses a property graph structure (nodes, edges, and properties) with spaces, containers, and views to organize and contextualize industrial data. Provides APIs for creating, ingesting, and querying models. | | [NEAT](/cdf/deploy/neat) | A solution for domain experts and developers that enables rapid development of data models. Handles ETL of data instances and ingests models and instances as knowledge graphs into CDF. Acts as an interface to the data modeling service outside of CDF. | | [Cognite Toolkit](/cdf/deploy/cdf_toolkit) | Tooling to manage, configure, and deploy resources in CDF. | ## Further reading * [Data modeling concepts](/cdf/dm/dm_concepts) – Core concepts: property graph, spaces, instances, containers, views, and data models. * [Data modeling guides](/cdf/dm/dm_guides) – Examples and best practices for extending CDM, performance, and CI/CD. * [Data modeling principles](/cdf/deploy/neat/data_modeling/principles) – Organizational principles and best practices for data modeling with NEAT. * [Resource naming conventions](/cdf/deploy/reference/cdf_resource_naming_conventions) – Complete reference for naming CDF resources. # Extending the core data model via GraphQL Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_extending_core_data_model Learn how to create custom data models that extend the Cognite core data model using the Data modeling language extension for GraphQL. Cognite's [core data model](/cdf/dm/dm_reference/dm_core_data_model) offers standardized building blocks for industrial data, forming the basis for more specialized models. This article outlines how to use the [Data modeling language extension for GraphQL (DML)](/cdf/dm/dm_graphql/dm_data_modeling_language) to create a **custom data model** that extends the core data model. ## Import types from the core data model By default, new custom data models don't have any type definitions. Before extending the core data model, you must import the type definitions you need from the core data model into the custom data model. To include the [CogniteSchedulable](/cdf/dm/dm_reference/dm_core_data_model#schedulable-props) feature from the core data model in your custom data model: Copy the type definition from the core data model to your custom data model: ```graphql wrap showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} interface CogniteSchedulable @container( indexes: [ { identifier: "startTime", fields: ["startTime"], cursorable: true } { identifier: "endTime", fields: ["endTime"], cursorable: true } { identifier: "scheduledStartTime" fields: ["scheduledStartTime"] cursorable: true } { identifier: "scheduledEndTime" fields: ["scheduledEndTime"] cursorable: true } ] ) { startTime: Timestamp endTime: Timestamp scheduledStartTime: Timestamp scheduledEndTime: Timestamp } ``` Specify which data model you want to import the **view definition** from. Use the `@import` directive to import a view definition from another data model. The `dataModel` argument of the `@import` directive is used to specify which data model you are importing the view definition from. When you publish a type with the `@import` directive your current data model will be updated to include the definition of the view as it is defined in the source data model, as opposed to defining a similar but separate view definition in your current data model. A view definition cannot contain both an `@import` directive with a dataModel argument and a `@view` directive at the same time as this can create referencial conflicts when importing the view definition. You need to remove any `@view` directives from the view definition when you add `@import` as described above or you will not be able to publish the updated model. For example, to import the `CogniteSchedulable` view from the core data model instead of creating a new view with the same schema in the custom data model: ```graphql wrap {18-25} showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} interface CogniteSchedulable @container( indexes: [ { identifier: "startTime", fields: ["startTime"], cursorable: true } { identifier: "endTime", fields: ["endTime"], cursorable: true } { identifier: "scheduledStartTime" fields: ["scheduledStartTime"] cursorable: true } { identifier: "scheduledEndTime" fields: ["scheduledEndTime"] cursorable: true } ] ) @import( dataModel: { externalId: "CogniteCore", version: "v1", space: "cdf_cdm" } ) { startTime: Timestamp endTime: Timestamp scheduledStartTime: Timestamp scheduledEndTime: Timestamp } ``` If you copy the type definition **without** the `@import` directive and publish the data model, you create a **new** view and container in your custom data model space **independent** from the core data model. You won't be able to access the data of the views stored in the core data model from your custom data model. Also, functionality such as path materialization of [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) won't work in your custom data model. ## Resolve transitive view dependencies When you import a type from another data model, you must also import all the views it depends on. To import a "Pump" view that has a direct relation to a "Location" view: Add the type definition for "Location" to your data model: ```graphql wrap showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Pump @import( dataModel: { externalId: "my-custom-datamodel" version: "v1" space: "my-custom-space" } ) { name: String location: Location } ``` The data model won't validate since the "Location" type isn't defined yet. To define it, you need to import the "Location" view in the same way you imported "Pump": ```graphql wrap {14-19} showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type Pump @import( dataModel: { externalId: "imaginary-datamodel" version: "v1" space: "imaginary-space" } ) { name: String location: Location } type Location @import( dataModel: { externalId: "my-custom-datamodel" version: "v1" space: "my-custom-datamodel" } ) { name: String } ``` ## Extend imported views When you've imported the necessary types from the core data model, you can extend them to match your needs. To extend the [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) concept from the core data model with a new `department` field: Import [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) into your data model: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type CogniteAsset implements CogniteVisualizable & CogniteDescribable & CogniteSourceable @view(space: "cdf_cdm", version: "v1") @import { # ... omitted CogniteAsset fields } ``` Define a `CustomCogniteAsset` type that implements the [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) type: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type CustomCogniteAsset implements CogniteAsset & CogniteVisualizable & CogniteDescribable & CogniteSourceable { # ... omitted CogniteAsset fields department: String } interface CogniteAsset implements CogniteVisualizable & CogniteDescribable & CogniteSourceable @view(space: "cdf_cdm", version: "v1") @import { # ... omitted CogniteAsset fields } ``` In the example above: * We removed the `@import` and the `@view` from `CustomCogniteAsset` because the type is owned by the custom data model and not imported from the core data model like the `CogniteAsset` type. * We changed `CogniteAsset` to be an interface instead of a type. The GraphQL specification requires this, and it has no impact on the functionality of the imported `CogniteAsset` type. * We included all the fields from the `CogniteAsset` on the `CustomCogniteAsset`. The GraphQL specification requires that all fields of the base interface are included when implementing it in another type. ## Override types for edges and direct relations When you have defined an extension to an existing type, you typically want any direct relation fields that reference the base type to reference the extension type instead. This example defines an extension of `CogniteAsset` called `CustomCogniteAsset`: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type CustomCogniteAsset implements CogniteAsset & CogniteVisualizable & CogniteDescribable & CogniteSourceable { # ... omitted CogniteAsset fields parent: CogniteAsset @mapping(property: "assetHierarchy_parent") } ``` The parent direct relation still references the base type (`CogniteAsset`). If you query for data on the parent direct relation via `CustomCogniteAsset`, you won't be able to access the data of the `department` field since it only exists in `CustomCogniteAsset`. To solve this issue, you need to manually override the type of the parent direct relation on the `CustomCogniteAsset` to reference the `CustomCogniteAsset` type instead of the base type: ```graphql wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type CustomCogniteAsset implements CogniteAsset & CogniteVisualizable & CogniteDescribable & CogniteSourceable { # ... omitted CogniteAsset fields parent: CustomCogniteAsset @mapping(property: "assetHierarchy_parent") } ``` You can only change the type of a field from the base type like this if you're overriding the existing type with a type that implements it. For example, you can't change the parent field type to [CogniteEquipment](/cdf/dm/dm_reference/dm_core_data_model#equipment) because [CogniteEquipment](/cdf/dm/dm_reference/dm_core_data_model#equipment) doesn't extend [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset). # Files in data modeling Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_integrate_files Learn how to use files with the core data model and integrate them with assets in Cognite Data Fusion. Cognite's [core data model](/cdf/dm/dm_reference/dm_core_data_model) uses [*CogniteFile*](/cdf/dm/dm_reference/dm_core_data_model#file) to represent files. The actual file *content* is handled by the [File content](/api-reference/concepts/20230101/file-content) API. This article describes how to use files with the core data model. ## CogniteFile in the core data model [CogniteFile](/cdf/dm/dm_reference/dm_core_data_model#file) uses the `assets` property to associate a file with one or more [CogniteAssets](/cdf/dm/dm_reference/dm_core_data_model#asset), and other properties to hold metadata of the file. The **access control** for CogniteFile is handled by the **datamodelinstances** access-control list (ACL). This is the only ACL required to access files in the core data model. ## Working with files using the Python SDK ### Create, retrieve and delete files To create, retrieve, and delete files using the Python SDK: To **create** a file, use the Cognite Python SDK `CogniteFileApply` data class: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from pathlib import Path from cognite.client.data_classes.data_modeling import NodeId from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteFileApply, CogniteFile # Create a file node in the specified space: client.data_modeling.instances.apply( CogniteFileApply(name="FileName", space="Space", external_id="File001") ) ``` ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Retrieve the created file node: file_info_retrieved = client.data_modeling.instances.retrieve_nodes( nodes=NodeId("Space", "File001"), node_cls=CogniteFile ) print(file_info_retrieved) ``` ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # When you have created the file node, to upload the content: client.files.upload_content("file.txt", instance_id=NodeId("Space", "File001")) # To download the content: client.files.download_to_path("downloaded_file.txt", instance_id=NodeId("Space", "File001")) ``` ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # To delete a file, delete the node: client.data_modeling.instances.delete(nodes=NodeId("Space", "File001")) ``` ### Associate files to CogniteAsset To associate files to CogniteAsset: Provide the instance identifiers of the asset nodes: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.data_modeling import DirectRelationReference, NodeId from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteFileApply, CogniteFile # Create or modify a file node with the asset references: client.data_modeling.instances.apply( CogniteFileApply( space="Space", name = "FileName", external_id="File001", assets=[ DirectRelationReference("Space", "Asset001"), DirectRelationReference("Space", "Asset002"), ], ) ) ``` ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Retrieve the file node to see the asset references: file_info_retrieved = client.data_modeling.instances.retrieve_nodes( nodes=NodeId("Space", "File001"), node_cls=CogniteFile ) print(file_info_retrieved) ``` ## Create and upload files using the API You can ingest files represented with CogniteFile directly through the API. Create a file node in the Data Modeling Service, request an upload link, and upload the file contents. You'll need to replace the placeholders in brackets `{}` used for the endpoint URLs and payload examples with appropriate values for your data model and files. To create and upload files using the API: **API documentation**: [Create or update nodes/edges](/api-reference/concepts/20230101/instances) **Endpoint URL**: `https://{cdf_cluster}.cognitedata.com/api/v1/projects/{cdf_project_name}/models/instances` **Payload example**: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "instanceType": "node", "space": "{instance_space}", "externalId": "{external_id}", "sources": [ { "source": { "type": "view", "space": "cdf_cdm", "externalId": "CogniteFile", "version": "v1" }, "properties": { "name": "{file_name}", "mimeType": "{mime_type}" } } ] } ] } ``` It's important that you include a `source` object mapping to the `CogniteFile` view in the list of `sources` in the payload. Without this, you'll receive an error when requesting an upload link in the next step, because the instance won't be able to be identified by the [File](/api-reference/concepts/20230101/files) service. If you've created a [custom view extending CogniteFile](/cdf/dm/dm_guides/dm_extending_core_data_model) that you want to target instead, however, replace the values for `space`, `externalId` and `version` in the `source` object with the corresponding values for your custom view: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "source": { "type": "view", "space": "{custom_space}", "externalId": "{CustomFileView}", "version": "{view_version}" } } ``` You aren't required to provide the `mimeType` property in the `properties` object, but it's useful for visibility in [Search](/cdf/explore/search) and for endpoints like [Engineering diagram detection](/api-reference/concepts/20230101/engineering-diagrams) to be able to process the file. **API documentation**: [POST /files/uploadlink](/api-reference/concepts/20230101/file-content) **Endpoint URL**: `https://{cdf_cluster}.cognitedata.com/api/v1/projects/{cdf_project_name}/files/uploadlink` **Payload example**: ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "instanceId": { "space": "{instance_space}", "externalId": "{external_id}" } } ] } ``` If you are uploading a file larger than 5 GiB you'll need to request a [POST /files/initmultipartupload](/api-reference/concepts/20230101/file-content) instead. Send a HTTP PUT request to the `uploadUrl` (or `uploadUrls` if you're using a multipart file upload link) in the response result from the previous query, with the relevant 'Content-Type' and 'Content-Length' headers according to the content of the file. Learn more: * [POST /files/uploadlink](/api-reference/concepts/20230101/file-content) * [POST /files/initmultipartupload](/api-reference/concepts/20230101/file-content) ## Related API reference Upload and download file content using instanceId. Create and manage file metadata in your data model. # Time series in data modeling Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_integrate_with_time_series Learn how to integrate time series with data modeling using CogniteTimeSeries and the Time Series API. For time series operations in data modeling: * **Datapoint operations** (insert, retrieve, query): See [Time series and datapoints](/api-reference/concepts/20230101/time-series-data) * **Metadata management** (create, update, delete time series): Use the [Instances API](/api-reference/concepts/20230101/instances) Cognite's core data model has a built-in CogniteTimeSeries concept to ingest and retrieve **time series**. CogniteTimeSeries integrates with the [Time Series API](/api-reference/concepts/20230101/time-series-data). To ingest and retrieve **data points**, you must use the same API. This article describes how time series are handled differently in data modeling and in asset-centric data models. ## Migrate existing time series To turn asset-centric time series into `CogniteTimeSeries` nodes in the Cognite core data model (`cdf_cdm`), use the Cognite Toolkit migration plugin. Run `cdf migrate timeseries`. By default, the command links each existing time series to a data modeling node and locks the original resource so later changes go through data modeling. See [Migrating files and time series](/cdf/deploy/cdf_toolkit/guides/plugins/migration_plugin/files_timeseries). ## Create time series In data modeling, each time series is a [node](/cdf/dm/dm_concepts/dm_property_graph#nodes) in the property graph. You can create, update, and delete time series using the same API as other [instances](/cdf/dm/dm_concepts/dm_spaces_instances#instance). The `instanceType` is `node`, and the combination of `space` and `externalId` identifies the time series and links it to the data points. Time series must be of a specific type: `numeric`, `string`, or `state` (*private beta*). You set the type when creating the time series. The type affects how the data points are stored, and to prevent potential data loss, you can't change it later. The `isStep` property determines how to interpolate between data points, and you can change the property later. Data modeling doesn't support all the time series properties of the asset-centric data model. ### Create time series with the API To create time series in data modeling, use the [create or update nodes/edges](/api-reference/concepts/20230101/instances) endpoint. The node must have data in the `cdf_cdm:CogniteTimeSeries/v1` system view to be recognized as a time series. The `type` is the only mandatory property. ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "space": "test", "externalId": "minimal_example", "instanceType": "node", "sources": [ { "source": { "type": "view", "space": "cdf_cdm", "externalId": "CogniteTimeSeries", "version": "v1" }, "properties": { "name": "Time series A", "type": "numeric" } } ] } ] } ``` ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "space": "test", "externalId": "hello", "instanceType": "node", "sources": [ { "source": { "type": "view", "space": "cdf_cdm", "externalId": "CogniteTimeSeries", "version": "v1" }, "properties": { "type": "numeric", "isStep": true, "unit": { "space": "cdf_cdm_units", "externalId": "temperature:deg_c" }, "sourceUnit": "C", "name": "Hello, world!", "description": "My new description", "tags": ["TagA", "TagB"], "aliases": ["hello_world"], "assets": [], "equipment": [], "sourceId": "examples", "sourceContext": "documentation", "sourceCreatedTime": "2024-08-28T13:16:25.228Z", "sourceUpdatedTime": "2024-08-28T13:20:25.228Z", "sourceCreatedUser": "John Doe", "sourceUpdatedUser": "Jane Doe" } } ] } ] } ``` ### Create time series with the Python SDK ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling import DirectRelationReference, NodeId from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteTimeSeriesApply # Instantiate Cognite SDK client: client = CogniteClient() # Insert a new CogniteTimeSeries client.data_modeling.instances.apply( CogniteTimeSeriesApply( space="test", external_id="hello", name="Hello, world!", is_step=True, time_series_type="numeric", unit=DirectRelationReference("cdf_cdm_units", "temperature:deg_c"), ) ) # Insert data points client.time_series.data.insert( instance_id=NodeId("test", "hello"), datapoints=[ (1724845953621, 0.0), (1724845970101, 1.0), ], ) # Retrieve data points client.time_series.data.retrieve( instance_id=NodeId("test", "hello"), start=0, ) ``` For more examples using the Python SDK, please see the [cognite-sdk documentation](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/time_series.html#). ## State time series (private beta) This feature is in [private preview](/cdf/product_feature_status#private-preview). For more information and to sign up, contact your Cognite representative. State time series represent discrete operational states of equipment, for instance, `ON`/`OFF`, `OPEN`/`CLOSED`, `RUNNING`/`IDLE`/`ERROR`. Unlike numeric or string time series, state time series have a predefined set of valid states and support specialized aggregations for analyzing operational patterns. ### Key differences between time series types | Aspect | Numeric | String | State | | - | - | - | - | | **Values** | Continuous measurements | Free-form text | Predefined state set | | **Aggregations** | min/max/avg/sum | Not supported | duration/transitions/count per state | | **Typical use case** | Sensor measurements | Labels/comments | Equipment operational states | ### Get started To use state time series: 1. Create a **state set** that defines valid states (integer, string pairs) using the `CogniteStateSet` view 2. Create a **state time series** by setting `type: "state"` and linking to the state set via the `stateSet` property. 3. Ingest state data points with numeric and/or string values. 4. Query with specialized aggregations: `stateCount`, `stateTransitions`, and `stateDuration`. For detailed API documentation, complete examples, constraints, and best practices, see [State time series](/dev/concepts/resource_types/state_timeseries). ## Synchronization from data modeling to the time series API When you create a `CogniteTimeSeries` in data modeling, it's automatically synchronized with the Time Series API (`/timeseries`). In the Time Series API, the time series you created in data modeling has an `instanceId` property (reusing the `externalId` property could cause conflicts with existing time series): ```json wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "instanceId": {"space": "...", "externalId": "..."}, // other time series properties: "id": ..., "externalId": ..., "name": ..., "metadata": ..., } ``` ### Updates The properties you can set in data modeling, can only be updated using the Data Modeling API. You can still update the other properties through the Time Series API: * `externalId` * `assetId` * `dataSetId` * `metadata` You can update a time series created in data modeling to also have an `externalId` property in addition to the `instanceId`. This can be useful in scenarios where some applications understand `externalId`, but not `instanceId`. ### Synchronization process A background "sync worker" ensures that time series created in data modeling is synchronized with the same time series seen using the Time Series API. #### Expected latency This synchronization process typically happens within seconds, but isn't *instantaneous/atomic*. During periods of high load, the process may take a few minutes. If you create a time series in data modeling and immediately query the Time Series API, the newly created time series may not be visible yet. #### Immediate consistency guarantees If the Time Series API receives an `instanceId` that it doesn't recognize, it triggers a synchronization for the instance to guarantee that the time series has been synced at least once, though it may not always be fully up to date. A delay can happen when you update an existing CogniteTimeSeries and, for example, change the `unit` property. The property may not be immediately updated in the Time Series API. You can create a time series in data modeling and immediately ingest data points into it. If the background syncer has not yet processed the new time series, the ingestion request triggers a synchronization, ensuring that the time series exists before proceeding. ### Deleting CogniteTimeSeries Time series created in data modeling can only be deleted using the data modeling API. Some time after the time series has been deleted from data modeling (typically within seconds), it is deleted in the Time Series API. If data points are written to the Time Series API after the instance has been deleted by the data modeling service, the data points may **appear** to be stored (write requests receive a `200 OK` response) until the time series in the Time Series API has been deleted. After this time, **all data will be deleted**. This may cause problems if you delete a time series instance and then recreate it. When a recreated time series uses the same `instanceId` as a deleted instance, data points for the newly created instance may get routed to the old time series. As a result, newly written data points may be missing from the new instance since they were deleted as part of the old time series deletion. When you have to delete and recreate a time series instance, always include a verification step. Example: Periodically send a `POST timeseries/byids` request to the Time Series API for the specific `instanceId` you have deleted, with the `ignoreUnknownIds` parameter set. When the Time Series API returns an empty response, it confirms that the instance has been deleted, and you can safely recreate the time series instance. For more information about the `/byids` API endpoint, see the [Time Series API documentation](/api-reference/concepts/20230101/time-series). This information applies to time series created using the data modeling API. It doesn't apply to time series created using the Time Series API. ## Access control Data modeling uses [spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space) for governance and [access control](/cdf/dm/dm_concepts/dm_access_control), **not** data sets. To create and read CogniteTimeSeries, you need the following capabilities: * `dataModelsAcl.READ` to the `cdf_cdm` space to read the core data model schema. * `dataModelsAcl.READ` to the `cdf_extraction_extensions` space to read properties set by Cognite extractors. * `dataModelInstancesAcl.READ` to the `cdf_cdm_units` space to use units. * `dataModelInstancesAcl.READ/WRITE/WRITE_PROPERTIES` to the space with the time series instances. If you have read/write access to a time series, you can also read/write its data points. If you modify `dataSetId` or `assetId` using the Time Series API, the time series and data points may be accessible to users without data modeling access. Users with access to `all` time series will also have access to time series in data modeling. Time series with a **write-protected** data set **denies** write access to data points, unless the user has `datasets:owner` access to the data set. Properties set by data modeling aren't write-protected. Time Series with an `instanceId` cannot have security categories. ## Related API reference Insert, retrieve, and query time series datapoints using instanceId. Combine time series with expressions and functions. Subscribe to changes in time series data points. Create and manage time series metadata in your data model. # Performance considerations Source: https://docs.cognite.com/cdf/dm/dm_guides/dm_performance_considerations Learn about key data modeling factors that impact performance when ingesting and querying data. This article outlines key data modeling factors that impact performance when ingesting and querying data. To manage containers and govern schema components, you can use [NEAT](/cdf/deploy/neat) (open source) and the [Cognite Toolkit](/cdf/deploy/cdf_toolkit). Because data modeling and query execution are tightly connected, understanding one often helps improve the other. [Debugging Query Performance](/cdf/dm/dm_guides/dm_debug_query_performance) introduces *debug notices*, a tool for analyzing concrete queries, helping you assess how well your data model supports real-world usage. ## Indexes and constraints To improve the speed of data retrieval and sorting, you can index any property defined in a container. However, indexes also come with a performance cost during certain operations. You can configure a maximum of [10 indexes for each container](/api-reference/concepts/20230101/containers). Each index can include an individual property, or combinations of properties ([composite indexes](#composite-indexes)). The **order** of the properties is important for composite indexes. The properties most **frequently** included in queries are primary candidates to include in an index. Also, you should consider the **size** of the data set, and the **update frequency** of the data. You can specify two main index types for the data modeling service: [Btree](#btree-indexes) and [inverted](#inverted-index) indexes. The examples in this section assume the following container definitions: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Example container definitions space: site_assets container: # A container for the "Asset" type with a single property "asset_id" - externalId: Asset name: Asset usedFor: node description: The Asset container properties: asset_id: name: asset_id type: type: int64 site: name: site type: type: text name: name: name type: type: text # [... more asset specific container properties could be included here] # A container for the "Description" type - externalId: Description usedFor: node properties: title: type: type: text list: false collation: ucs_basic nullable: true description_text: type: type: text list: false collation: ucs_basic nullable: true labels: type: type: text list: true collation: ucs_basic nullable: true ``` ### Why not index all properties? Well-planned indexes that match the query patterns of your data consumers improve the performance of data retrieval operations. But indexes come at a cost, and creating an index on the wrong properties can be detrimental to both data retrieval, and insert/update operations: * Indexes built on properties containing a lot of unique data will often impede performance. * BTree indexes for properties with large amounts of data can use significant amounts of memory and can cause intermittent error/timeout statuses for update or add operations. * Composite indexes for millions of pieces of data can lead to poor performance during add and update operations for the indexed properties. It may impact the experienced stability of your solutions and your data objects. Having an index for every property is unlikely to give the desired result even if the [total property count in the container would allow it](/api-reference/concepts/20230101/containers). Too many indexes, across too many properties, in too many related containers may lead to slow performance during ingestion, query, search, and filtering operations. Slow performance can result in a lot of push-back on the client in the form of retry statuses from the data modeling service. ### BTree indexes BTree indexes maintain data in a sorted order and lets you quickly look up instances based on the properties defined in the index. The indexes are ideal for queries using both **range** and **equality** filters. Conceptually, a BTree is similar to a phone book, which is sorted by name, allowing you to find a person's phone number quickly. BTree indexes are particularly useful for data stored as **scalar data types**. To see the scalar data types available in container properties, navigate to the `type` > `PrimitiveProperty` section of the [Create or update containers](/api-reference/concepts/20230101/containers) entry in the API documentation. The following example illustrates creating BTree indexes for the asset name and site: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} indexes: siteIndex: properties: - site indexType: btree cursorable: true nameIndex: properties: - name indexType: btree ``` ### Inverted indexes Inverted indexes allow you to filter on the values of array properties efficiently. Conceptually, this is like a cookbook where you can quickly find all recipes that contain a specific ingredient. Inverted indexes cannot be made cursorable. This example adds a single inverted index for the labels property in the asset container: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} indexes: labels: properties: - labels indexType: inverted ``` ### Composite indexes Composite indexes are effective when properties are used together in query conditions, for example, a `WHERE` filter using several related properties. Build composite indexes for **properties that are commonly used together** to filter or sort data. This improves performance because the data modeling service data store quickly can find the data that meet all criteria included in the index. The following query examples illustrate the usefulness of a composite inverted index, and the order of the properties in the index: * a) return assets where their name starts with the letter A, from site X * b) return all assets at site Y * c) return all assets where their name starts with the letter B Query *a)* and *b)* should see a performance benefit by defining the composite inverted index described as `site_asset_name` in the [the example below](#composite-index-examples). However, the `site_asset_name` won't improve the performance of query *c)*. To improve the performance for query *c)*, you would need the `asset_name_site` composite index [below](#composite-index-examples). The `asset_name_site` index would also improve performance for query *a)*, but not for query *b)*. You cannot create composite indexes using properties in different containers We suggest you use the requirements collection phase to test, and validate, your data volume expectations for the container(s). Also use it to identify the most likely query patterns. Then build the smallest number of composite indexes you need, so your graph will deliver the performance and scalability your consumers expect during data onboarding and consumption. With large data sets, try using distinct and small composite indexes to support the specific queries you expect consumers will use. #### Composite index examples ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Composite index combining the site and asset name for the Asset container indexes: site_asset_name: # An inverted index for the combined site and asset name properties should deliver more performant queries # when searching for asset names within a site. properties: - site - name # alternative definition for properties included in the index: # properties: [site, name] indexType: inverted asset_name_site: # An inverted index for the combined asset name and site properties should deliver more performant queries # when searching for whether a site is linked to a (set of) asset name(s). properties: - name - site # alternative definition for properties included in the index: # properties: [name, site] indexType: inverted ``` ### Pagination (cursorable indexes) Use cursorable indexes to support efficient **pagination** through large data sets. For example, when your users need to receive all available data based on a set of filters. To enable these types of indexes, set the `cursorable:` parameter to `true` for a BTree index. A cursorable index will have an internal unique identifier padded to it, and cursors have a suitable tiebreaker when the indexed values are not unique. As a result, cursorable indexes are more expensive than non-cursorable indexes, so use them carefully. For incremental processing — "*Return all data that has changed, based on this/these filters*" — use the [sync](/api-reference/concepts/20230101/instances) feature. For small or static data sets, we recommend that you keep the default value for `cursorable:` (`false`). Inverted indexes can't be cursorable. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # A "cursorable" btree index for the asset_id property in the example Asset container indexes: # The BTree index for the asset_id property in the example Asset container assetIdIndex: properties: - asset_id # alternative definition for properties included in the index: # properties: [asset_id] indexType: btree cursorable: true # Set to true to enable a cursorable index ``` For DMS's query executor to use the cursorable index, the sort in your query *must* match the cursorable index. A cursorable index is ordered. If you have a cursorable index on `property1` and `property2`, its data could looks like this: | property1 | property2 | | - | - | | 1 | 1 | | 1 | 2 | | 2 | 1 | | 2 | null | | null | null | To cursor through this index efficiently, you need to either sort ascending with nulls last: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - property: property1 direction: ascending nullsFirst: false - property: property2 direction: ascending nullsFirst: false ``` or descending with nulls first: ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - property: property1 direction: descending nullsFirst: true - property: property2 direction: descending nullsFirst: true ```

Example: Pagination by cursorable index. To query for all **books** by their **author** and **title**: 1. Add an index to the **author** and **title** properties in the corresponding container. ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} indexes: author_title: properties: - author - title indexType: btree cursorable: true ``` 2. In the query, specify that you want to sort your results by the same properties. ```yaml {7-13} theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} query: with: books: nodes: filter: # removed for briefness sort: - property: author direction: ascending nullsFirst: false - property: title direction: ascending nullsFirst: false select: books: # removed for briefness ``` During runtime, the system will have the capability to use the index for pagination. Sorting on the indexed properties *must* be in the **with** clause of the query.
### Indexed base properties These base properties are indexed with B-Tree indexes by default: * `space`, `externalId` * `type` (cursorable) * `startNode` (**not cursorable**) * `endNode` (**not cursorable**) This implies that: * Cursoring through all instances based on `space` or `type` is performant. * `startNode` and `endNode` filters are efficient, but not suitable for cursoring. * Avoid filters on other base properties, such as `lastUpdatedTime` and `createdTime`, except on very small amounts of data. ### Indexes on direct relations Traversing direct relations does a lookup on the identifier on the nodes in either end. This requires B-Tree indexes to be efficient. If you, for example, have two pumps in a system of pumps: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {'theme':'base', 'themeVariables': { 'lineColor':'#8b949e'}}}%% graph LR; Pump1-->System1 Pump2-->System1 ``` To retrieve the system for `Pump1`, the query executor needs to do a lookup in the core node container for the node with the internal identifier corresponding to `System1`. This internal identifier property has a B-Tree index on it to make this lookup efficient. If, however, you want to retrieve all pumps in `System1`, the query executor need to do a lookup in the `Pump.system` direct relation property. To make this query performant, you need to add a B-Tree index to `Pump.system`. ### Constraints See [Containers, views, polymorphism, data models](/cdf/dm/dm_concepts/dm_containers_views_datamodels#constraints) for more information about constraints, and how to create them for your containers. Adding constraints creates indexes for the properties you include in the constraint definition and affect performance when you add or update data (to `asset_id` in the example). The reason is that CDF needs to make sure that the value you're adding or updating remains unique for the `asset_id` property (in the example). ```yaml wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Example constraint definition for asset_id in the Asset container constraints: asset_id_is_unique: constraintType: uniqueness properties: - asset_id ``` ## Query performance considerations See the sections below for query performance considerations. Refer to the [Debugging Query Performance](/cdf/dm/dm_guides/dm_debug_query_performance) article for information about how to debug query performance issues. ### Excessive joins slow down queries When querying instances, you can use the `sources` field to specify which containers or views to retrieve data from. You can also use `hasData` filters to return only instances with data in certain containers or views. Also, you can use `nested` filters to filter on the properties of a node pointed to by a direct relation. For all these operations, the query executor needs to join in the underlying containers. For example, if you select data from a view that maps five containers, the query executor needs to perform five joins to fetch all the data. For every nested filter, the query executor must perform another two joins. The number of joins quickly adds up to significantly reduce the query performance. To keep your queries performant, limit the number of containers and views you query from and the number of nested filters. Also, consider carefully how you use **[`hasData`](/cdf/dm/dm_concepts/dm_querying#HasData-filter)** filters. ### `/list` endpoint pitfalls The default sort on the list endpoint is by the internal ID. The index on the internal ID is cursorable, and it's not well-suited to use with arbitrary filters. The only filters that'll be performant with the default sort are `space` filters and `hasData` filters. To use the list endpoint with other filters, add a cursorable index on the properties you want to filter on. Make sure the sort in your query matches the index. The `/list` endpoint emits a cursor regardless of the sort, but not all sorts will be performant. If you have more than a few thousand instances, your sort *must* be backed by a cursorable index. ## Container sizing How large (wide), or small (narrow), should the containers be when you build your schema? The answer to this seemingly simple question is, frustratingly; "It depends". The challenge is two-fold: 1. **We cannot exclusively use tiny containers**, and; 2. **We cannot exclusively use huge containers** It might be very convenient for a property to be its own container. However, if that worked well we would not need the containers concept in the first place. On the other hand, it could also be convenient to construct a single container with all the properties described by your information model. Or the properties needed by all consumers of data from your CDF project. But, again we would not need containers in the first place. Note that neither of these challenges imply that you can't ever use tiny, or large containers! It does imply that you fully understand the trade-offs you face when using large, or small, containers. To design a performant container, you need to consider questions like: * How will your consumers use data hosted in the container(s)? * How many instances and properties are needed for a search/query/filter operation? * Which properties are indexed, and using what kind of index? * Which constraints apply? * Which, and how many, views are combining data from more than one container? * How will the views access data from the container(s)? * Will the data in the container be updated together with other data in other containers? For example, creating a wide container with many indexed properties, and a frequently updated column like `last_inspection` or `last_viewed` (properties) will probably lead to poor performance. The same goes for frequently updated properties where the data is actually changed whenever the property is updated. These kinds of containers are prime candidates for you to split to "small" containers. ### The content of a CDF data modeling container How you design a container shapes the "physical" storage of the data in the data modeling services data store. The data store is a PostgreSQL database instance. When you use the data modeling service API to create a container, think of the container as a table in PostgreSQL. The table contains all specified properties *and* the additional **internal properties** the service needs. For example, if you create a container with a single property named `my-property`, the container will look like this in the data modeling data store: | *internal\_data* | *internal\_data* | *space* | *my-property* | | - | - | - | - | | YYY | | 10 | foo | | | XXX | 10 | not foo | Notice the internal data columns — **properties** — in the example. The data modeling service hides them and their data from you, but uses them extensively behind the scenes. The underlying data store doesn't allow composite indexes for properties that are stored in different containers. ### When is a container too big? A container is too big (wide) when it has **too many properties and indexes** to be performant. Wide containers with many properties means that ingesting or updating data takes longer and the indexes can become bloated. This affects write and query performance for the data consumer and can cause more [HTTP status 408](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/408) errors. Each CDF project comes with a set of preconfigured [default limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#schema-and-instance-limits). One of the limits is the maximum number of properties per schema object. By default, this limit is 100 properties, but it **can be increased**. In addition to **consumption** and **data update** patterns, the "correct" width of a container depends on the **number of other wide containers** in the graph. In some cases, it may be efficient to have a container with 200 properties. If only a few, or none, of the properties have relationships, or dependencies, that must be updated when you onboard or update data, and if you don't have consumption relationships to and from other containers, having 10 wide containers (>50 properties) can also be a good solution. To safeguard the performance of your solution, make sure that you experiment with, and **test the scalability** for, your onboarding and consumption patterns. Don't use the data modeling service as an "[entity-attribute-value](#eav-model)" (EAV) store. Learn why here: [Anti-pattern: the entity-attribute-value (EAV) model](#eav-model) ### When is a container too small? Using only small containers results in **excessive query-time JOINs** as the data modeling service processes queries from your data consumers. This, in turn, makes it difficult to form useful queries and return relevant data to the consumer. In addition, the **overhead of internal index structures** will likely result in poor performance. The data modeling service manages indexes for many of the internal properties and composite indexes for combinations of internal properties. The service uses the indexes to speed up access for an instance when it receives an explicit `instance_id` to retrieve data from. The indexes are also used when a consumer enumerates everything in the container, with or without using a space filter. These enumerations are common when executing list operations using the `has_data` filter(s). Another side effect of using a "one container per property" approach is that you can't specify indexes consisting of multiple properties (a composite index). This reduces performance across the joined tables when consumers query for data. There are, however, patterns where it may be efficient to use small containers. For example: * You want to store data that's independent, or infrequently used together with data from other containers. * You're hosting a small amount of data. * Your data is frequently updated. Example: The container has a `last_viewed` property, or a `last_updated` property. ### Properties for filtering vs fetching properties In the data modeling service, a query has two "phases": 1. For any per-table expression, the service determines **which instances** to return in **what order**. 2. The service fetches the requested data from the containers representing those instances. For this, the data modeling service has two strategies: 1. Fetch all the data from the container. 2. Query a single system-internal container; `node_source` and `edge_source`. This system-internal container holds JSON blob representations of the data stored in all the other containers. The first strategy reflects how the schema and data modeling services work today. The strategy works well for queries that need to return specific data from a few specific containers. However, the strategy doesn't work well when you have a query that returns *all* the data for any number of instances. If you, for example, have 10,000 containers, the first strategy would need to look in 10,000 tables for every instance — not a workable solution. Instead, the strategy searches through a table, with a (sub-)query equivalent to: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} select * from node_source join the_nodes_we_need_to_return using(internal_data, internal_data) order by .... ``` This operation can be slow, and will often result in timeouts and error conditions for the consumer. ### The cost of updates In most transactional systems, an update to data typically creates new copies of the data. It will likely not change any existing data "in place". As an example, if you have a container with `document_body` and `last_viewed` properties and have configured the system to update the `last_viewed` property frequently, each update also saves the `document_body`. Also, any update to data including at least one indexed column results in updates to **all** indexes for that table. ([See technical details](https://www.cybertec-postgresql.com/en/hot-updates-in-postgresql-for-better-performance).) In sum, these updates contribute to container and index bloat that will affect performance for data consumers (clients). We recommend that you: * **Don't include frequently updated data in wide containers.** Examples of properties like these include data for `view_count`, `last_viewed`, `last_updated`, `reactions`, etc. * **Keep rarely changed and heavily indexed data in separate containers** from frequently updated and indexed property data. ### The cost of queries In general, fewer JOINs result in the best query performance. A query only joins containers that your data consumers include by using filters. The database **query optimizer** uses **statistics** to decide the order of the filters and JOINs. The data store maintains the statistics and hosts it per container. These statistical values are based on **sampling**, and not continuous recording. As you combine more properties or containers, the statistics used by the optimizer are less likely to support good decision-making. For instance, when the optimizer determines the order of JOINs, it estimates the expected number of instances (rows) in the container. If the statistics for the container confirm that it's processing for a small container, this is considered a strong signal that the container has few instances (rows). The **size of the table** is a simple statistic to maintain. Knowing much about the number of instances and the size of the container, is a lot more difficult. This is especially the case for data where properties `a`, `b`, `c`, `d`, and `e` are `not null` for a specific row, but for many others they are `null`. This may, for example, be the case for a sparsely populated container with many properties, but relatively little data in total. This statistics problem impairs the precision of the optimizer. In some cases, it may result in the system needing to scan all the container data to produce the data to return. The data modeling service isn't optimized for full scans of containers hosting a lot of data for many properties. As a result, consumers may receive `HTTP status code 408` after submitting a query to the data modeling service. ### Share data containers This example illustrates how you can create a performant data filtering pattern with a combination of properties, their composite index, and a `WHERE` filter. This example assumes that you have three properties containing data: `a`, `b`, and `c`: 1. Create a **composite index** concatenating the property values in the same order as you want to structure a `WHERE` filter in a query: `index(a, b, c)`. 2. Add the **filtering** clause `WHERE a = $1 AND b = $2 AND c >= $3` to your query. `$1`, `$2`, and `$3` are **parameters**. Before the query is sent to the data modeling service, the parameters are replaced with actual values to test for in the `WHERE` clause. However, you can't build a composite index if the properties are **from separate containers**, for example if `(a, b)` are in a different container from `c`. For such cases, you can move `(a, b)` to the same container as `c` (or the other way around), or you can repeat/duplicate property `c` and its data to several different containers. Use the information in this section to **test and verify** the best option for your data and your consumers. An **index scan** determines which rows to return as matching your filter. Filter-less list/sync queries always use this method to return results. If you split properties across different containers, and the properties are indexed separately, the data modeling service optimizer can choose different strategies. The service uses statistical analysis and is likely to assume that `c >= $3` only matches a few instances (rows). Consequently, it'll look up those rows first. Next, it verifies that the identified rows match `(a, b)`. For the match, the system doesn't always use the composite `(a, b)` index you have defined. If property `c` represents a `last_modified` timestamp property for an instance, it assumes that the property is frequently updated as is the case with most `last_modified` properties. When the query attempts to filter on `last_modified >= 0`, or `c >= 0` in the example above, it's not efficient to begin the comparison by processing the `c` filter. The filter probably matches a lot of data. Also, for a platform like CDF, much of the data will have changed at about the same time. Additionally, the filter probably identifies less than *all* data content for the container. If so, the potential matches of both indexes — `(a, b)`, and `(c)` — might be exhausted. The data modeling service then uses the resulting bitmaps to intersect, looking for candidate rows. This is a necessary, but low-performing strategy. Client applications may see slow query responses or timeout errors (HTTP status 408) when they use the filter. To **summarize**, if a property is frequently used by filters and the filters span data across many containers, it may be beneficial to duplicate the data for the property in multiple containers. One such example is the default `space` property. We observed that our consumers often filter `space`. The property was originally protected, but we decided to add the space information to every container created. You can now add the `space` property to any index definition. Rather than keeping the property and related information in a single container, we duplicate the small amount of data represented by the space identifier. This helps you achieve better performance for queries if you need to filter or query together with the `space` property from any container. ## Modeling anti-pattern: the Entity-Attribute-Value (EAV) model The EAV model is an **anti-pattern** and we strongly recommend **not** using it when building containers in CDF. The Entity-Attribute-Value (EAV) model is often used in data stores where low-execution-cost flexibility is the most important outcome. Rather than using conventional structured tables with fixed columns, the EAV model uses three main components: **entities**, **attributes**, and **values**. One of the appealing aspects of the EAV approach is its ability to handle dynamic and sparse data. You can add new attributes without altering the underlying schema. This makes the pattern highly adaptable to evolving data requirements. However, we strongly recommend **NOT** using the EAV model with the CDF data modeling service. Using the model will have a **significant negative impact** on cost and performance. Some key problems are: * **Instance explosion**. Each new property you add to the EAV store multiplies the nodes and instances in the system. * **Increased storage cost**. Adding only two properties can increase your stored data by more than 600%. This is mostly due to additional instance-related system metadata. The **storage and processing cost** is exacerbated with more properties, not amortized. * **Increased query processing**. With EAV instances, every query results in large matrices of **many-to-many-to-many JOINs**. The JOINs are based on tables where "everything is mixed together", which also undermines the service's ability to collect and use meaningful statistics to determine data distribution. The **indexes** can't be granular. Instead, they have to include everything, also properties that don't need indexing. That causes the indexes to be far less efficient, more expensive to maintain, and each property far more expensive to query. The **query execution** engines will be running blind, impacting the performance of the solution, and likely causing timeouts (HTTP status 408). Instead of using the EAV model, we recommend that you establish schemas using **first class properties**. If properties are always used together in a lot of filters, and they effectively filter the data, you can include them in a composite index to enable fast lookup operations. The query engine will have statistics for the properties to reason about and to run the query. Typically, it **does not** JOIN every possible combination of edges-to-`property`-nodes. **Everything is not a first-class property**. We've seen that most data models follow a [power/Zipf distribution](https://www.britannica.com/topic/Zipfs-law). The data in the most popular property is used 1/10th of the time. The data in the second most popular property is used 1/20th of the time. This pattern continues for each subsequent property. For this reason, not all data stored by the service needs to use property typed properties. If you don't plan on using a property for filtering, sorting, or indexing, you can save it in a JSON-object with JSON-typed properties. That allows the service to store data and do string-like filtering without you having to define schemas for everything stored in the data model(s). # Data modeling examples and best practices Source: https://docs.cognite.com/cdf/dm/dm_guides/index Explore examples and best practices for working with data modeling in Cognite Data Fusion, from building hierarchies to optimizing performance. Learn how to model data to support natural language queries and enable Atlas AI agents across domains and workflows. Learn best practices for designing scalable data models in CDF, including layered architecture, design principles, and performance optimization. Build an asset hierarchy with the Cognite core data model and populate it with transformations. Create custom data models by extending the core data model using GraphQL data modeling language. Connect files to your data model instances using the Cognite core data model. Link time series data to your data model instances for operational insights. Understand key factors that impact performance when ingesting and querying data. Use debug notices to analyze and optimize your data model queries. Learn why query latency can vary and how to design applications that tolerate occasional slow requests. Delete nodes that don't have associated container data using a Python script. Deploying data models with CI/CD is a Cognite Toolkit workflow. See [Deploying with the Cognite Toolkit](/cdf/deploy/cdf_toolkit) in the Deploy tab. # Handling latency variability in query APIs Source: https://docs.cognite.com/cdf/dm/dm_guides/latency_variability Learn why data modeling query APIs can have outlier response times and how to design SDKs and applications that remain reliable when occasional slow queries occur. Latency in distributed systems is rarely constant. When you query instances in data modeling, most requests complete quickly, but a small percentage can take significantly longer. This pattern is called **latency variability**. ## Why it matters Latency outliers can impact user experience and system reliability even when median latency is low. If you assume every request completes quickly, a small number of slow queries can cause retries, thread exhaustion, cascading timeouts, and noisy alerts. Designing for variability helps your application stay responsive and predictable at scale. ## Where latency variability shows up In data modeling, latency variability typically shows up when you query instances with these endpoints: * `/models/instances/list` * `/models/instances/query` (including GraphQL) Latency variability usually shows up as a “long-tail” pattern in latency percentiles: * **p50 (median)** latency is low. * **p90/p95** latency is higher. * A small percentage of requests (**p99**) can be much slower. For example, you might see: * p50: 200 ms * p90: 1.5 s * p99: 4.5 s This does not mean most queries are slow. It means most queries are fast, and a few outliers take longer due to query and backend execution conditions. ## Why latency can vary Latency for `/list` and `/query` depends on several factors. ### Query shape and index alignment Filters and sorting that align with **defined indexes** are typically much more efficient than queries that require scans or expensive sorting. To learn how indexes affect query execution, see [Performance considerations](/cdf/dm/dm_guides/dm_performance_considerations). ### Schema and view complexity Wider views and more complex schemas can increase the amount of data the service needs to materialize and process. For example: * Views mapping many containers * Queries that traverse relations (direct or reverse) * Large property selection sets Unused views and view versions can increase tail latency. The longest tail latencies often occur during full project schema reloads after cache expiration, when the service reconciles all views and properties. Cleaning up unused views and view versions can reduce these outliers. ### Data volume and distribution The number of instances and how values are distributed affects the cost of query execution. Two queries with the same structure can behave differently if the data distribution changes over time. ### Payload size Large responses (for example, hundreds of KB to multiple MB) increase total time due to: * Serialization on the server * Network transfer If you only need a subset of fields, use property selectors in the `select` section to return only the properties you need. This reduces payload size and avoids wasteful transfer. ### Backend execution conditions Even when you run the same query repeatedly, internal conditions can change: * Cache state (cold vs. warm) * Execution plan preparation * Resource scheduling and load distribution These factors can occasionally increase latency for individual requests. ## What to expect * **Occasional slow requests (outliers)** are expected. * Outliers do **not** indicate service degradation if: * Requests complete successfully (HTTP 200). * You don’t see a sustained latency increase across most requests. CDF provides availability guarantees, but not fixed per-request latency guarantees for individual endpoints. ## How to design for latency variability Design your SDKs and applications so they remain reliable when a small number of requests are slow. ### Don’t assume constant response time Avoid designs that require every call to complete within a strict time budget (for example, “always under 0.5 seconds”). ### Use timeouts appropriate to context Set client-side timeouts that match the user interaction context: * For interactive UX, keep timeouts shorter and show progress or partial results. * For background jobs, allow longer timeouts and add retries. ## Further reading * [Performance considerations](/cdf/dm/dm_guides/dm_performance_considerations) * [Debugging query performance](/cdf/dm/dm_guides/dm_debug_query_performance) * [Cognite Databricks integration documentation](https://cognite-cognite-databricks.readthedocs-hosted.com/en/latest/) # Cognite data models Source: https://docs.cognite.com/cdf/dm/dm_reference/cognite_data_models Learn about Cognite's out-of-the-box data models, including the core data model and process industry data model. Cognite provides out-of-the-box data models to get you started building a structured, flexible, and contextualized **knowledge graph** of your industrial data. Cognite's core data model offers standardized building blocks for industrial data, forming the basis for more specialized models. Cognite's process industries data model extend the core data model to cater to specific process industry requirements. # Core data model Source: https://docs.cognite.com/cdf/dm/dm_reference/dm_core_data_model Explore Cognite's core data model with standardized building blocks for industrial data, including CogniteAsset, core features, and core concepts. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). Cognite's core data model—**CogniteCore**—offers standardized building blocks for industrial data, forming the basis for more specialized models. It's available from the **`cdf_cdm`** [system space](/cdf/dm/dm_concepts/dm_system_schemas). The core data model uses two methods to define properties: [core features](#core-features) and [core concepts](#core-concepts): * The **core features** are sets of properties that are available for **many** concepts. * The **core concepts** have properties that are specific to a **single** concept. Each core feature and core concept comes with a matching [**container**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) and [**view**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views). The exceptions are stateless concepts, which only have a view and no container. The **containers** have efficient [indexes](/cdf/dm/dm_concepts/dm_containers_views_datamodels#indexes) and constraints and can be used at scale out-of-the-box. The **views** combine the properties of the core features and the core concepts. The core data model can be **extended**, currently via [GraphQL](/cdf/dm/dm_guides/dm_extending_core_data_model). For example, Cognite's **industrial data models** extend the **core data model** to meet common, industry-specific needs. ## Access control In addition to the `dataModelInstancesAcl` capabilities to read/write the instances, users of the core data model need the following capabilities: * `dataModelsAcl.READ` to `cdf_cdm` to read the core data model schema. * `dataModelsAcl.READ` to `cdf_extraction_extensions` to read properties set by Cognite extractors. * `dataModelInstancesAcl.READ` to `cdf_cdm_units` to use units. For more information, see [Control access to a graph](/cdf/dm/dm_concepts/dm_access_control). ## Core features This section collects commonly used properties, with common definitions. If a core concept uses a different definition, or narrows the scope of, an otherwise common property, it will be called out in the section that defines the core concept. ### CogniteDescribable The CogniteDescribable core feature has properties to create names and descriptions for concepts. | Property | Type | Description | | - | - | - | | name | String | The name of the instance. | | description | String | The description of the instance. | | tags | List of strings | A list of tags for the instance. Max: 1000 | | aliases | List of strings | Alternative names for the instance. | ### CogniteSourceable The CogniteSourceable core feature has properties to represent how the source information in concepts is stored. | Property | Type | Description | | - | - | - | | sourceId | String | The identifier in the source system. | | sourceContext | String | The context of the `sourceId`. For systems where the `sourceId` is globally unique, leave `sourceContext` empty. | | source | Direct relation to a [source system](#sourcesystem-props) | A direct relation to a source system instance. | | sourceCreatedTime | Timestamp | The time the instance was created in the source system (if available). | | sourceUpdatedTime | Timestamp | The time the instance was updated in the source system (if available). | | sourceCreatedUser | String | The user ID for the user who created the instance in the source system. It may or may not be the same as the CDF user ID. | | sourceUpdatedUser | String | The user ID for the user who updated the instance in the source system. It may or may not be the same as the CDF user ID. | ### CogniteSourceSystem The CogniteSourceSystem core feature has properties to represent the source system where the information is stored. | Property | Type | Description | | - | - | - | | version | String | The version identifier for the source system. | | manufacturer | String | The manufacturer of the source system. | The following core features are also available for this core feature: [CogniteDescribable](#describable-props) ### CogniteSchedulable The CogniteSchedulable core feature has properties to represent start and end times for a concept, for example an activity. | Property | Type | Description | | - | - | - | | startTime | Timestamp | The start time of an activity (or similar). | | endTime | Timestamp | The end time of an activity (or similar). | | scheduledStartTime | Timestamp | The planned start time of an activity (or similar). | | scheduledEndTime | Timestamp | The planned end time of an activity (or similar). | ### CogniteVisualizable The CogniteVisualizable core feature has properties to enable a standard display of objects and to represent a reference to a 3D resource. | Property | Type | Description | | - | - | - | | object3D | Direct relation to [Cognite3DObject](#cognite3dobject) | The related Cognite3DObject instance representing the 3D resource. | ## Core concepts ### CogniteUnit The CogniteUnit concept has properties to represent a single unit of measurement, for example Ampere. | Property | Type | Description | | - | - | - | | symbol | String | The symbol for the unit of measurement. | | quantity | Enum | The physical quantity the unit measures. | | source | String | The source of the unit specification. | | sourceReference | String | A reference to the source of the unit specification. Typically, a URL to a standard, or to documentation of the source. | The following core features are also available for this concept: [CogniteDescribable](#describable-props) ### CogniteAsset The CogniteAsset concept has properties to represent systems that support industrial functions or processes. Assets are often called "functional locations". You can structure assets hierarchically according to your organization's criteria. For example, an asset can represent a **water pump** that's part of a larger **pump station** asset at a **clarification plant** asset. | Property | Type | Description | | - | - | - | | parent | Direct relation | A direct relation to the parent of the asset. | | children | Reverse direct relation | A virtual reverse relation to assets with this asset as their parent. It is resolved at query time and isn't stored or materialized in the container. | | root | Direct relation | An automatically updated reference to the top-level asset of the hierarchy. | | path | List of direct relations | An automatically updated ordered list of this asset's ancestors, starting with the root asset. Enables subtree filtering to find all assets under a parent. | | pathLastUpdatedTime | Timestamp | The last time `path` was updated for this asset. | | equipment | Reverse direct relation list to [CogniteEquipment](#equipment) | An automatically updated list of equipment related to the asset. | | timeSeries | Reverse direct relation list to [CogniteTimeseries](#timeseries) | An automatically updated list of [time series](#timeseries) related to the asset. | | assetClass | Direct relation to [CogniteAssetClass](#assetclass) | Specifies the class of the asset, for example **Pump** or **Heat exchanger**. | | type | Direct relation to [CogniteAssetType](#assettype) | Specifies the type of the asset, for example **Centrifugal pump** or **Air cooled heat exchanger**. | | activities | Reverse direct relation list to [CogniteActivity](#activity) | An automatically updated list of [activities](#activity) related to the asset. | | files | Reverse direct relation list to [CogniteFile](#file) | An automatically updated list of [files](#file) related to the asset. | For hierarchies, `root`, `path`, and `pathLastUpdatedTime` are system-maintained hierarchy properties. `children` is a virtual reverse relation, so its behavior differs from these fields. Depending on your SDK-generated model classes, `children` might not be available as a standard typed property. Any `path` values longer than 100 are truncated to the first 100 items. Assets with more than 100 ancestors in the hierarchy will not get a complete `path`. The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props), [CogniteVisualizable](#visualizable-props) #### CogniteAssetClass The CogniteAssetClass concept has properties to represent the class of an asset, for example **Pump** or **Heat exchanger**. | Property | Type | Description | | - | - | - | | code | String | A unique identifier for the class of asset. | | standard | String | A text string to specify which standard the class is from. | The following core features are also available for this concept: [CogniteDescribable](#describable-props) #### CogniteAssetType The CogniteAssetType concept has properties to represent the type of an asset, for example **Centrifugal pump** or **Air cooled heat exchanger**. | Property | Type | Description | | - | - | - | | code | String | A unique identifier for the type of asset. | | standard | String | A text string to specify which standard the type is from. | | assetClass | Direct relation to [CogniteAssetClass](#assetclass) | Specifies the class the type belongs to. | The following core features are also available for this concept: [CogniteDescribable](#describable-props) ### CogniteEquipment The CogniteEquipment concept has properties to represent physical supplies or devices. Example: a **pump** or a **valve**. | Property | Type | Description | | - | - | - | | asset | Direct relation to [CogniteAsset](#asset) | The asset the equipment is related to. | | activities | Reverse direct relation list to [CogniteActivity](#activity) | An automatically updated list of [activities](#activity) related to the equipment. | | timeSeries | Reverse direct relation list to [CogniteTimeseries](#timeseries) | An automatically updated list of [time series](#timeseries) related to the equipment. | | serialNumber | String | The serial number of the equipment. | | manufacturer | String | The manufacturer of the equipment. | | equipmentType | Direct relation to [CogniteEquipmentType](#equipmenttype) | Specifies the type of the equipment. | | files | List of direct relations to [CogniteFile](#file) | A list of files the equipment relates to. For example, documentation and data sheet that belong to this equipment. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props) #### CogniteEquipmentType The CogniteEquipmentType concept has properties to represent types of equipment, for example **centrifugal pumps**. | Property | Type | Description | | - | - | - | | code | String | A **unique** identifier for the type of equipment. | | equipmentClass | String | The class of equipment. | | standard | String | An identifier for the standard this equipment type is sourced from, for example, ISO14224. | | standardReference | String | A reference to the source of the equipment standard. | The following core features are also available for this concept: [CogniteDescribable](#describable-props) ### CogniteFile The CogniteFile concept has properties to represent files. | Property | Type | Description | | - | - | - | | assets | List of direct relations to [CogniteAssets](#asset) | A list of assets this file is related to. | | equipment | Reverse direct relation list to [CogniteEquipment](#equipment) | An automatically updated list of equipment this file is related to. | | mimeType | String | The MIME type of the file. | | directory | String | Contains the path elements from the source (if the source system has a file system hierarchy or similar). | | isUploaded | Boolean | Specifies if the file content has been uploaded to Cognite Data Fusion or not. | | uploadedTime | Timestamp | The time the file upload completed. | | category | Direct relation to [CogniteFileCategory](#filecategory) | Specifies the detected category the file belongs to. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props) #### CogniteFileCategory The CogniteFileCategory concept has properties to represent the categories of files as determined by contextualization or categorization. | Property | Type | Description | | - | - | - | | code | String | An identifier for the category, for example, `AA` for Accounting (from Norsok.) | | standard | String | The name of the standard the category originates from, for example `Norsok`. | | standardReference | String | A reference to the source of the category standard. | The following common properties are also available for this concept: [CogniteDescribable](#describable-props) ### CogniteActivity The CogniteActivity concept has properties to represent activities. Activities typically happen over a period and have a start and end time. | Property | Type | Description | | - | - | - | | assets | List of direct relations to [CogniteAsset](#asset) | A list of assets this activity is related to. | | equipment | List of direct relations to [CogniteEquipment](#equipment) | A list of equipment this activity is related to. | | timeSeries | List of direct relations to [CogniteTimeseries](#timeseries) | A list of time series this activity is related to. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props), [CogniteSchedulable](#schedulable-props) ### CogniteTimeSeries The CogniteTimeSeries concept has properties to represent a series of data points in time order, for example the temperature readings from a water pump. | Property | Type | Description | | - | - | - | | isStep | Boolean | Specifies whether the time series is a **step** time series or not. | | type | Enum | Specifies the data type of the data points. Valid values: `numeric`, `string`, `state`. | | sourceUnit | String | The unit specified in the source system. | | unit | Direct relation to [CogniteUnit](#unit) | The unit of the time series. | | stateSet | Direct relation to [CogniteStateSet](#stateset) | The state set for state time series. Defines the valid states (integer, string pairs) for the time series. Only applicable when type is `state`. | | assets | List of direct relations to related [CogniteAsset](#asset) | A list of assets the time series is related to. | | equipment | List of direct relations to [CogniteEquipment](#equipment) | A list of equipment the time series is related to. | | activities | Reverse list of direct relations to [CogniteActivity](#activity) (and subtypes) | An automatically updated list of activities the time series is related to. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props) ### CogniteStateSet The CogniteStateSet concept has properties to represent a set of valid states for state time series. Each state consists of a numeric value (integer) and a string value pair, for example `[(0, "OFF"), (1, "ON")]`. | Property | Type | Description | | - | - | - | | states | List of state value pairs | A list of valid states. Each state contains `numericValue` (32-bit integer) and `stringValue` (up to 255 UTF-8 characters). Maximum 100 states. | **State constraints:** * Both numeric and string values must be unique within a state set. * Numeric values must be between -2,147,483,648 and 2,147,483,647. * String values can be up to 255 UTF-8 characters. The following core features are also available for this concept: [CogniteDescribable](#describable-props) ### CogniteAnnotation The CogniteAnnotation concept has properties to represent annotations from contextualization results or links. | Property | Type | Description | | - | - | - | | confidence | float64 | The confidence that this is a good match. | | status | Enum | The status of the annotation. Valid values: `Suggested`, `Approved`, and `Rejected`. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props) #### CogniteDiagramAnnotation The CogniteDiagramAnnotation concept has properties to represent an annotation for a diagram. | Property | Type | Description | | - | - | - | | startNodePageNumber | int64 | The number of the page on which the annotation is located in the `startNode` file. The first page has number 1. | | endNodePageNumber | int64 | The number of the page on which the annotation is located in the `endNode` file if an `endNode` is present. The first page has number 1. | | startNodeXMin | float64 | A value between \[0,1]. The minimum abscissa of the bounding box (left edge). Must be strictly less than `startNodeXMax`. | | startNodeXMax | float64 | A value between \[0,1]. The maximum abscissa of the bounding box (right edge). Must be strictly more than `startNodeXMin`. | | startNodeYMin | float64 | A value between \[0,1]. The minimum ordinate of the bounding box (bottom edge). Must be strictly less than `startNodeYMax`. | | startNodeYMax | float64 | A value between \[0,1]. The maximum ordinate of the bounding box (top edge). Must be strictly more than `startNodeYMin`. | | startNodeText | String | The text extracted from within the bounding box on the `startNode`. | | endNodeXMin | float64 | A value between \[0,1]. The minimum abscissa of the bounding box (left edge). Must be strictly less than `endNodeXMax`. Only applicable if an `endNode` is defined. | | endNodeXMax | float64 | A value between \[0,1]. The maximum abscissa of the bounding box (right edge). Must be strictly more than `endNodeXMin`. Only applicable if an `endNode` is defined. | | endNodeYMin | float64 | A value between \[0,1]. The minimum ordinate of the bounding box (bottom edge). Must be strictly less than `endNodeYMax`. Only applicable if an `endNode` is defined. | | endNodeYMax | float64 | A value between \[0,1]. The maximum ordinate of the bounding box (top edge). Must be strictly more than `endNodeYMin`. Only applicable if an `endNode` is defined. | | endNodeText | String | The text extracted from within the bounding box on the `endNode`. Only applicable if an `endNode` is defined. | The following core features are also available for this concept: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props), [CogniteAnnotation](#annotation) ## 3D concepts #### Cognite3DTransformation Defines a comprehensive 3D transformation, which positions an object in 3D space using translation, rotation (Euler angles), and scale. | Property | Type | Description | | - | - | - | | translationX | Float | Displacement along the X-axis. | | translationY | Float | Displacement along the Y-axis. | | translationZ | Float | Displacement along the Z-axis. | | eulerRotationX | Float | Rotation around the X-axis (radians). | | eulerRotationY | Float | Rotation around the Y-axis (radians). | | eulerRotationZ | Float | Rotation around the Z-axis (radians). | | scaleX | Float | Scale factor along the X-axis. | | scaleY | Float | Scale factor along the Y-axis. | | scaleZ | Float | Scale factor along the Z-axis. | #### CogniteCubeMap Defines the cube map projections surrounding a 3D entity, such as `Cognite360Image`. | Property | Type | Description | | - | - | - | | front | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the front projection of the cube map. | | back | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the back projection of the cube map. | | left | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the left projection of the cube map. | | right | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the right projection of the cube map. | | top | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the top projection of the cube map. | | bottom | Direct relation to a [CogniteFile](#file) | A direct relation to a file holding the bottom projection of the cube map. | ### Base 3D concepts #### Cognite3DModel Groups 3D data into a single model entity. | Property | Type | Description | | - | - | - | | type | Enum | The 3D model type. Valid values: `CAD`, `PointCloud`, or `Image360`. | | thumbnail | Direct relation to [CogniteFile](#file) | Thumbnail image for the 3D model. | The following core feature is also available for this concept: [CogniteDescribable](#describable-props). #### Cognite3DRevision Represents revision data for 3D models. Typically accessed through [CognitePointCloudRevision](#cognitepointcloudrevision), [Cognite360ImageCollection](#cognite360imagecollection-view), or [CogniteCADRevision](#cognitecadrevision). | Property | Type | Description | | - | - | - | | status | Enum | The status of the revision. Valid values: `Queued`, `Processing`, `Done`, or `Failed`. | | published | Boolean | Specifies if the revision is published or not. | | type | Enum | The type of 3D model this is a revision of. Valid values: `CAD`, `PointCloud`, or `Image360`. | | model3D | Direct relation to [Cognite3DModel](#cognite3dmodel) | The 3D model this is a revision of. | | sourceFile | Direct relation to [CogniteFile](#file) | The source file used to create this revision. | #### Cognite3DObject Represents the virtual position of a physical object, connecting an asset to one or more 3D resources. | Property | Type | Description | | - | - | - | | asset | Reverse direct relation to [CogniteAsset](#asset) | The asset referencing this 3D object. | | cadNodes | Reverse direct relation list to [CogniteCADNode](#cognitecadnode) | A list of maximum 1000 CAD nodes representing the connected asset. | | pointCloudVolumes | Reverse direct relation list to [CognitePointCloudVolume](#cognitepointcloudvolume) | A list of maximum 1000 point cloud volumes representing the connected asset. | | images360 | Edge | Edge connection to Cognite360Image annotations representing the connected CogniteAsset. | | xMin | Float | Lowest X value in the bounding box. | | xMax | Float | Highest X value in the bounding box. | | yMin | Float | Lowest Y value in the bounding box. | | yMax | Float | Highest Y value in the bounding box. | | zMin | Float | Lowest Z value in the bounding box. | | zMax | Float | Highest Z value in the bounding box. | The following core feature is also available for this concept: [CogniteDescribable](#describable-props). #### Cognite3DGroup Groups related 3D entities together. | Property | Type | Description | | - | - | - | | groupType | Enum | The type of group. Valid value: `Station360` | ### CAD #### CogniteCADModel (view) This view filters [Cognite3DModel](#cognite3dmodel) where `type` equals `CAD`. It's used for navigating across `CogniteCADModel` instances. | Property | Type | Description | | - | - | - | | revisions | Reverse direct relation list to [CogniteCADRevision](#cognitecadrevision) | A list of revisions for this CAD model. | This view inherits properties from the [Cognite3DModel](#cognite3dmodel) concept. The following core feature is also available: [CogniteDescribable](#describable-props). #### CogniteCADRevision Represents revision data for CAD models. | Property | Type | Description | | - | - | - | | revisionId | int64 | 3D API revision identifier for this CAD model. | | model3D | Direct relation to [CogniteCADModel](#CogniteCADModel) | A direct relation to a [CogniteCADModel](#CogniteCADModel) instance. | This concept extends [Cognite3DRevision](#cognite3drevision). #### CogniteCADNode Represents contextualized nodes from a CAD model. | Property | Type | Description | | - | - | - | | object3D | Direct relation to [Cognite3DObject](#cognite3dobject) | Direct relation to the 3D object grouping for this node. | | model3D | Direct relation to [Cognite3DModel](#cognite3dmodel) | Direct relation to [Cognite3DModel](#cognite3dmodel) ([CogniteCADModel](#cognitecadmodel) in view version). | | cadNodeReference | String | Reference to a node within a CAD model. | | revisions | List of direct relations to [Cognite3DRevision](#cognite3drevision) | Direct relations to revision instances which this `CogniteCADNode` exists in. | | treeIndexes | List of int64 | Tree indexes in the same order as revisions. Used by Reveal and similar applications to map from `CogniteCADNode` to a tree index entry in the backing 3D model. | | subTreeSizes | List of int64 | Subtree sizes in the same order as revisions. Used by Reveal and similar applications to determine how many nodes exist below this node in the hierarchy. | The following core feature is also available for this concept: [CogniteDescribable](#describable-props). ### 360° images #### Cognite360ImageModel (view) This view filters [Cognite3DModel](#cognite3dmodel) where `type` equals `Image360`. It's used for navigating across `Cognite360ImageModel` instances. | Property | Type | Description | | - | - | - | | collections | Reverse direct relation list to [Cognite360ImageCollection](#cognite360imagecollection-view) | List of image collections for this model. | This view inherits properties from the [Cognite3DModel](#cognite3dmodel) concept. The following core feature is also available: [CogniteDescribable](#describable-props). #### Cognite360ImageCollection (view) This view represents a logical collection of [Cognite360Image](#cognite360image) instances. It filters [Cognite3DRevision](#cognite3drevision) where `type` equals `Image360`. | Property | Type | Description | | - | - | - | | model3D | Direct relation to [Cognite360ImageModel](#cognite360imagemodel-view) | A direct relation to a `Cognite360ImageModel` instance. | This view inherits properties from the [Cognite3DRevision](#cognite3drevision) concept. The following core feature is also available: [CogniteDescribable](#describable-props). #### Cognite360Image Represents a 360° image capture. | Property | Type | Description | | - | - | - | | collection360 | Direct relation to [Cognite360ImageCollection](#cognite360imagecollection-view) | Direct relation to the image collection this image belongs to. | | station360 | Direct relation to [Cognite3DGroup](#cognite3dgroup) | Direct relation to the [Cognite3DGroup](#cognite3dgroup) instance that groups different `Cognite360Image` instances to the same station. | | takenAt | Timestamp | The time when the 6 photos were taken. | This concept includes properties from the following 3D concepts: [Cognite3DTransformation](#cognite3dtransformation), [CogniteCubeMap](#cognitecubemap). #### Cognite360ImageStation (view) This view groups images across collections to create a visual scan history. It filters [Cognite3DGroup](#cognite3dgroup) where `groupType` equals `Station360`. | Property | Type | Description | | - | - | - | | groupType | Enum | The type of group. Valid value: `Station360` | This view inherits properties from the [Cognite3DGroup](#cognite3dgroup) concept. The following core feature is also available: [CogniteDescribable](#describable-props). #### Cognite360ImageAnnotation This concept defines edges connecting [Cognite360Image](#cognite360image) to [Cognite3DObject](#cognite3dobject). The status is always `approved`. | Property | Type | Description | | - | - | - | | polygon | List of floats | A list of floats representing the polygon. The format depends on `formatVersion`. | | formatVersion | String | The storage format version for the polygon. | This concept extends [CogniteAnnotation](#annotation). The following core features are also available: [CogniteDescribable](#describable-props), [CogniteSourceable](#sourceable-props). ### Point cloud #### CognitePointCloudModel (view) This view filters [Cognite3DModel](#cognite3dmodel) where `type` equals `PointCloud`. It's used for navigating across `CognitePointCloudModel` instances. | Property | Type | Description | | - | - | - | | revisions | Reverse direct relation list to [CognitePointCloudRevision](#cognitepointcloudrevision) | A list of revisions for this point cloud model. | This view inherits properties from the [Cognite3DModel](#cognite3dmodel) concept. The following core feature is also available: [CogniteDescribable](#describable-props). #### CognitePointCloudRevision Represents revision data for point cloud models. | Property | Type | Description | | - | - | - | | revisionId | int64 | Revision identifier of the point cloud model. | | model3D | Direct relation to [CognitePointCloudModel](#cognitepointcloudmodel-view-only) | A direct relation to a [CognitePointCloudModel](#cognitepointcloudmodel-view-only) instance. | This concept extends [Cognite3DRevision](#cognite3drevision). #### CognitePointCloudVolume Defines a volume within a point cloud. | Property | Type | Description | | - | - | - | | object3D | Direct relation to [Cognite3DObject](#cognite3dobject) | Relation to the 3D object grouping for this volume. | | model3D | Direct relation to [Cognite3DModel](#cognite3dmodel) | Relation to the parent model. | | volumeReferences | List of strings | Volume metric hashes for 3D data storage access. | | revisions | List of direct relations to [Cognite3DRevision](#cognite3drevision) | Relations to revisions containing this volume. | | volumeType | Enum | The type of volume. Valid values: `Cylinder` or `Box`. | | volume | List of floats | 9 floats defining the volume coordinates. | | formatVersion | String | Version of the `volume` field format. | The following core feature is also available for this concept: [CogniteDescribable](#describable-props). # Limits and restrictions Source: https://docs.cognite.com/cdf/dm/dm_reference/dm_limits_and_restrictions Complete reference for data modeling, records, and streams resource limits in your CDF project. Understanding these limits helps you plan capacity for data modeling and records in your CDF project. Limits are grouped by area: * **Data modeling** — schema objects, instances, property values, instance query limits, and reserved identifiers * **Records and streams** — stream counts, template limits, and on-request capacity * **API concurrency** — simultaneous instance operations * **Monitoring** — view usage and request higher limits ## Data modeling Default limits for schema resources and instances. Your project may have different values depending on configuration. ### Schema and instance limits The table lists the **default** limits for data modeling schema resources and instances. | Name | Default limit | | - | - | | **Spaces** | 100 | | **Containers** | 1,000 | | - Container properties (total) | 25,000 | | - Container properties (per container) | 100 | | - Container properties (per container, records only) | 1,000 | | - Values in an enum container property | 32 | | **Views** | | | - View versions (total) | 2,000 | | - View versions (per view) | 100 | | - View properties (per view) | 300 | | - View properties (per view, records only) | 1,000 | | - Implemented views (per view) | 10 | | **Data models** | | | - Data model versions (total) | 500 | | - Data model versions (per data model) | 100 | | - Views (per data model) | 100 | | **Instances** | | | Live instances | 5,000,000 | | Soft deleted instances | 10,000,000 | ### Property value limits These limits apply to the values of container properties: | Data type | Maximum size | | - | - | | `text` properties | 128,000 bytes (UTF-8) | | `json` properties | 40,960 bytes (UTF-8) | | List properties | See below. | #### List size limits By default, list properties have these size constraints: * **Direct relations**: maximum 100 items * **Other types**: maximum 1,000 items You can customize the limits by setting the `maxListSize` parameter on container properties. You can only increase (not decrease) the `maxListSize` parameter after creating a container property. The maximum values for `maxListSize` are: * 600 for `int32` properties with a b-tree index. * 300 for `int64` properties with a b-tree index. * 2,000 for all other property types. ### Instance query limits On an instance query, `targetUnits` must contain between 1 and 10 values. This applies to the data modeling API. Omit `targetUnits` to skip unit conversion. A request outside this range fails with `targetUnits size must be between 1 and 10`. ### Reserved values The following values are reserved for Cognite internal use and cannot be used in your data models. #### Views and containers **Reserved `externalId` values**: * `Boolean`, `Date`, `Float32`, `Float64`, `Float`, `Int32`, `Int64`, `Int` * `JSONObject`, `Mutation`, `Numeric`, `PageInfo`, `Query`, `String` * `Subscription`, `TimeSeries`, `File`, `Sequence`, `Timestamp` **Reserved `properties` values**: * `createdTime`, `deletedTime`, `edge_id`, `extensions`, `externalId` * `lastUpdatedTime`, `node_id`, `project_id`, `property_group` * `seq`, `space`, `version`, `tg_table_name`, `startNode`, `endNode` **Reserved enum value names**: * `"true"`, `"false"`, `"null"` #### Spaces **Reserved `space` values**: * `cdf`, `dms`, `edge`, `node`, `pg3`, `shared`, `space`, `mapping`, `system` * `constraint`, `edge_source`, `history`, `identifier_by_space`, `index` * `node_source`, `project`, `property_group_constraint`, `property_group_index` * `property_group`, `property`, `version_info` ## Records and streams This section lists resource limits for records and streams. For an introduction to the concepts, see [About records and streams](/cdf/dm/records/concepts/records_and_streams). For sizing guidance and how the limits shape your design, see [Choosing and sizing streams](/cdf/dm/records/guides/choose_and_size_streams). ### Stream limits per project The table lists the **default** stream limits for a CDF project. The limits can be different for your CDF project depending on your specific configuration. | Limit | Default | | - | - | | Total active streams | 3 | | Active streams from `BasicLiveData` template | 1 | | Active streams from `BasicArchive` template | 2 | | Active streams from `ImmutableTestStream` template | 3 | | Soft-deleted streams | 30 | You can mix streams from different templates within the total active stream cap. To raise these limits, see [Request additional capacity](/cdf/dm/records/guides/choose_and_size_streams#request-additional-capacity). ### Stream template limits **Stream templates are in beta** Stream templates aren't in [production](/cdf/product_feature_status#production) yet. We may add templates, adjust limit values, or expose more options when you create a stream. Any changes apply only to **new** streams. Streams you've already created keep the limits that were in effect at creation time. For capacity planning and template selection, see [Choosing and sizing streams](/cdf/dm/records/guides/choose_and_size_streams). The stream template you select fixes the stream's limits at creation time. You can't change a stream's template or limits after you create it. The table lists the **default** limits for each stream template. | Limit | `BasicArchive` | `BasicLiveData` | `ImmutableTestStream` | | - | - | - | - | | Mutability | Immutable | Mutable | Immutable | | Max records per stream | 50 million | 5 million | 50 million | | Max data volume per stream | 50 GB | 15 GB | 50 GB | | Records ingested per 10 minutes | 170,000 | 170,000 | 800,000 | | Records updated or deleted per 10 minutes | Not supported | 85,000 | Not supported | | Ingestion throughput per 10 minutes | 170 MB | 170 MB | 1.5 GB | | Read throughput per 10 minutes | 500 MB | 500 MB | 1.5 GB | | Max unique properties per stream | 1,000 | 1,000 | 1,000 | | Max `lastUpdatedTime` filter interval | 365 days | No limit | 7 days | | Data retention | Unlimited | No time-based retention | 7 days | | Soft-delete recovery window | 6 weeks | 6 weeks | 1 day | | Max active streams per project | 2 | 1 | 3 | For what each template is designed for and how to select one, see [Stream templates](/cdf/dm/records/concepts/records_and_streams#stream-templates) and [Choose a default template](/cdf/dm/records/guides/choose_and_size_streams#choose-a-default-template). ### On-request capacity For enterprise customers with larger workloads, Cognite can enable higher limits on request. This is not automatic with your subscription. For the request process, see [Request additional capacity](/cdf/dm/records/guides/choose_and_size_streams#request-additional-capacity). For solution architecture estimates, Cognite has provisioned: | Capacity | Immutable streams | Mutable streams | | - | - | - | | Max records per stream | 5 billion | 100 million | | Max data volume per stream | 5 TB | 300 GB | Custom data retention periods and additional active streams are also available on request. Routine requests typically take 1 to 2 weeks. If your workload exceeds these figures, contact your Cognite representative to discuss your needs on a case-by-case basis. No API is available today to list which stream templates are enabled for your project. Contact your Cognite representative to confirm what's available beyond the defaults. Rate and concurrency limits for the Records and Streams API endpoints are documented in [API rate limits](/api-reference/concepts/20230101/rate-limits). ## API concurrency limits Concurrency limits control how many data modeling instance API operations can run simultaneously on your CDF project. The table lists the **default** concurrency limits for a CDF project. The limits can be different for your CDF project depending on your specific configuration. | Operation type | Default concurrency limit | | - | - | | Instance apply operations | 4 | | Instance delete operations | 2 | | Instance query operations | 8 | Transformations can use a maximum of 75% of the budget while the remaining 25% is reserved for other clients. Concurrency limits are applied project-wide, not per client. If a request exceeds any of the concurrency limits, it will be throttled with a `429 Too Many Requests` response. Learn more about [API rate limits](/api-reference/concepts/20230101/rate-limits). If you are using official Cognite SDKs, throttled requests are automatically retried with exponential backoff. If you are making a request without using an official SDK, make sure that you are using an appropriate retry strategy to handle 429 responses. ## Monitoring resource usage To view the current limits and usage statistics for your CDF project: In the Data fusion workspace, select Data models. On the Data models page, under Storage, select See all to view your project-specific limits and current usage statistics. ### Retrieve limits programmatically To retrieve limits and usage statistics programmatically: * Use the [DM statistics endpoint](/api-reference/concepts/20230101/dm-statistics) for project-wide statistics. * Use the [DM space statistics endpoint](/api-reference/concepts/20230101/dm-statistics) for space-specific statistics. ## Need higher limits? If you need increased limits for data modeling resources or instances, contact [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new). For records and streams capacity above the defaults, see [On-request capacity](#on-request-capacity) or [Request additional capacity](/cdf/dm/records/guides/choose_and_size_streams#request-additional-capacity). # Process industries data model Source: https://docs.cognite.com/cdf/dm/dm_reference/dm_process_industry_data_model Explore Cognite's process industries data model that extends the core data model for process industry requirements. Cognite's process industries data model—**CogniteProcessIndustries**—extends the [core data model](/cdf/dm/dm_reference/dm_core_data_model) to cater to specific requirements for the process industries. It's available from the **`cdf_idm`** [system space](/cdf/dm/dm_concepts/dm_system_schemas). ## Process industries concepts The extended process industries concepts come with matching [**containers**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) and [**views**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views). ### CogniteMaintenanceOrder The CogniteMaintenanceOrder concept has properties to represent **maintenance orders**. A maintenance order is a request or directive to perform maintenance tasks such as repair, inspection, or servicing of equipment or assets. It's often called a 'work order'. | Property | Type | Description | | - | - | - | | mainAsset | Direct relation to [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) | The primary asset (functional location) the maintenance order refers to. | | type | String | The code for the type of maintenance order, for example, 'PM01' for 'Corrective maintenance'. | | status | String | The lifecycle status of the maintenance order, for example, 'Partially complete'. It's often derived from the 'SystemStatus' field in SAP. | | priority | Int64 | A number that shows how important this maintenance order is. Lower numbers mean it's more important, and 0 means it's the most important. | | operations | Reverse direct relation list | Lists the individual work items on the maintenance order. An automatically populated list of operations related to the maintenance order. | The following core features and concepts are also available for this concept: [CogniteDescribable](/cdf/dm/dm_reference/dm_core_data_model#describable-props), [CogniteSourceable](/cdf/dm/dm_reference/dm_core_data_model#sourceable-props), [CogniteSchedulable](/cdf/dm/dm_reference/dm_core_data_model#scheduleable-props), [CogniteActivity](/cdf/dm/dm_reference/dm_core_data_model#activity) ### CogniteOperation The CogniteOperation concept has properties to represent **operations**, a specific part of the work included in a maintenance order. Often called a "work order item". | Property | Type | Description | | - | - | - | | maintenanceOrder | Direct relation to [CogniteMaintenanceOrder](#cognitemaintenanceorder) | The maintenance order the operation is related to. | | mainAsset | Direct relation to [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) | The primary asset (functional location) the operation is related to. | | phase | String | The code for the phase the operation belongs to. Each operation belongs to a single phase, for example, 'Scaffolding'. | | status | String | The lifecycle status of the operation, for example, 'Partially complete'. It's often derived from the `UserStatus` field in SAP. | | sequence | Int64 | A number that shows the order in which the operations are performed. Lower numbers mean they happen earlier, and 0 means it's the first operation. | | mainDiscipline | String | The primary skill or resource required to perform the operation, for example, "maintenance technician". | | numberOfMainDiscipline | Int64 | The number of resources required to perform the operation, for example, the number of required maintenance technicians. | | personHours | Float64 | The number of hours required to perform the maintenance operation. The duration is calculated by dividing the `personHours` by the `numberOfMainDiscipline`. | The following core features and concepts are also available for this concept: [CogniteDescribable](/cdf/dm/dm_reference/dm_core_data_model#describable-props), [CogniteSourceable](/cdf/dm/dm_reference/dm_core_data_model#sourceable-props), [CogniteSchedulable](/cdf/dm/dm_reference/dm_core_data_model#scheduleable-props), [CogniteActivity](/cdf/dm/dm_reference/dm_core_data_model#activity) ### CogniteNotification The CogniteNotification concept has properties to represent maintenance **notifications**. Notifications are formal records to report maintenance issues, defects, or requests, starting the process for planning and running maintenance activities. | Property | Type | Description | | - | - | - | | maintenanceOrder | Direct relation to [CogniteMaintenanceOrder](#cognitemaintenanceorder) | The maintenance order that was created from the notification. | | asset | Direct relation to [CogniteAsset](/cdf/dm/dm_reference/dm_core_data_model#asset) | The asset (functional location) the notification is related to. | | type | String | The code for the notification type, for example, "M1" for "Maintenance request". | | status | String | The current lifecycle status of the notification, for example, "order created". | | priority | Int64 | A number that shows how important the notification is. Lower numbers mean it's more important, and 0 means it's the most important. | | priorityDescription | String | Describes the priority of the notification. | The following core features and concepts are also available for this concept: [CogniteDescribable](/cdf/dm/dm_reference/dm_core_data_model#describable-props), [CogniteSourceable](/cdf/dm/dm_reference/dm_core_data_model#sourceable-props), [CogniteSchedulable](/cdf/dm/dm_reference/dm_core_data_model#scheduleable-props) # About data modeling Source: https://docs.cognite.com/cdf/dm/index Build expressive and scalable data models in Cognite Data Fusion (CDF) to create a structured knowledge graph of your industrial data, including assets, hierarchies, and equipment. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). A data model is an abstract representation of real-world entities. It organizes data elements and their properties and standardizes how they relate. Learn about the data modeling service (DMS) and core concepts like containers, views, spaces, and instances. Follow recommendations for composing and deploying data models and queries. Learn how to integrate time series data with your data model. Learn how to integrate files with your data model. Explore the Cognite core and industry data models, and understand system limits. Learn about high-performance storage for bulk structured data using records and streams. Use GraphQL to query, mutate, and define data models in Cognite Data Fusion. Design and validate data models in Excel before deploying them through the Cognite Toolkit. Part of the Deploy tab. Learn about the limits and restrictions for data modeling resources in your CDF project. REST endpoints for Spaces, Instances, Containers, Views, and Data models. # About records and streams Source: https://docs.cognite.com/cdf/dm/records/concepts/records_and_streams Understand records, streams, and record views for high-performance storage of bulk structured data in the industrial knowledge graph, including when to use them and how record views separate storage from consumption. When you're working with high-volume industrial data, storing each entry as a node in your knowledge graph becomes impractical. The graph becomes cluttered, query performance degrades, and you quickly hit instance budget limits. Records solve this problem by providing high-performance storage for bulk structured data that doesn't need to be part of the graph structure. Records are structured data objects stored in streams, separate from the industrial knowledge graph. Like data modeling instances, records use [record containers](#record-containers) to define storage schemas, [record views](#record-views) as consumption interfaces, and [spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space) for access control. Records don't create nodes or edges in the graph, which enables you to store billions of records without affecting graph performance or consuming instance budgets. After reading this article, you'll understand when to use records versus data modeling instances, how streams organize and manage your data, how record views separate storage from consumption, and how records integrate with the industrial knowledge graph. ## When to use records Records are designed primarily for **immutable, high-volume data** that doesn't require complex graph relationships. The service is optimized for write-once, read-many scenarios where data volume and analytics are critical. Common use cases include: * **High-volume immutable data**: Logs, events, and notifications (OPC UA events, PI EventFrames, well logs, manufacturing batch logs) * **Archived and historical data**: Completed work orders, resolved alarms, concluded activities * **Data with a defined lifecycle**: Active work orders or alarms that need updates during their lifecycle before being archived to immutable storage If your data requires frequent updates and will never be archived, consider using [data modeling nodes](/cdf/dm/dm_concepts/dm_spaces_instances#instance) instead. Nodes are better suited for persistent, frequently-changing entities in your knowledge graph. To decide whether records fit your workload and how much stream capacity you need, see [Choosing and sizing streams](/cdf/dm/records/guides/choose_and_size_streams). To understand when to choose records versus data modeling instances, see the [comparison table](#records-vs-nodes) below for detailed differences. ## Core concepts To work effectively with records, you need to understand these key concepts: * [Streams](#streams) define the lifecycle and performance characteristics of your data. * [Records](#records) are the individual data objects you store. * [Spaces](#spaces) provide access control and organization. * [Record containers](#record-containers) define the schema structure. * [Record views](#record-views) define a versioned interface for reading and writing records. ### Streams Streams are logical containers that organize your records and define how they behave throughout their lifecycle. When you create a stream, you choose a template that sets policies for: * **Retention periods**: How long records are stored before automatic deletion * **Mutability**: Whether records can be updated after ingestion * **Performance characteristics**: Ingestion and query throughput limits Each stream is created from one of the available [stream templates](#stream-templates), and this template cannot be changed after creating the stream. Streams are independent from spaces and containers. A single stream can contain records from multiple spaces with different container schemas, giving you flexibility in how you organize your data. **Using one stream across multiple solutions.** Because containers define schemas and spaces define access boundaries, a single stream can serve many solutions at the same time. Each solution can define its own containers and spaces and write into a shared stream, while the stream itself defines lifecycle, retention, and performance tier rather than solution isolation. Plan stream assignments around lifecycle and throughput needs, not around solution boundaries. Before you can ingest records, you must create a stream to hold them. When you query records, you reference the specific stream where they're stored. #### Stream templates **Stream templates are in beta** Stream templates aren't in [production](/cdf/product_feature_status#production) yet. For the current limit values of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). When you create a stream, select a stream template that defines the stream's behavior, performance, and lifecycle policies. Keep in mind these important considerations: * The choice between mutable and immutable records has significant scale implications. Different stream templates support different maximum record counts and storage capacities. * You cannot change a stream's template after creation, so choose your template carefully for production use. * Review [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits) before creating your streams, and select based on your expected data volume and mutability requirements. **Immutable streams** These templates are for data that should not be modified after ingestion. * **`BasicArchive`** is intended for **perpetual data storage**. Records are kept until you delete the stream. However, overall data volume is limited, so plan usage accordingly. * **`ImmutableTestStream`** is **exclusively for experimentation**. It's configured for high throughput and total data volume but has short data retention. Low retention in a soft-deleted state means you can quickly discard such streams when you no longer need them or recreate them to remove experimental data. **Note:** Never use the `ImmutableTestStream` template for production purposes. As this template allows significant load on the system, if we detect improper usage patterns, we may change the settings of streams created from this template as a last resort. **Mutable streams** These templates allow data updates but have lower total record limits compared to immutable streams. * **`BasicLiveData`** is intended for the **production mutable working set** (records that still change). It offers significant throughput but a low total record count compared to immutable templates. For the limit values of each template — maximum record counts, data volumes, throughput, retention, and soft-delete windows — see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). For how consumed properties accumulate and how to plan around the limit, see [Max unique properties](/cdf/dm/records/guides/choose_and_size_streams#max-unique-properties) in Choosing and sizing streams. The budget is consumed per container the first time you write a property through it, so containers in different spaces count separately even when their schemas are identical. Referencing containers or views in queries doesn't consume budget. For enterprise customers with larger workloads, we can enable higher per-stream limits on request. This is not automatic with your subscription. For the available capacity, see [On-request capacity](/cdf/dm/dm_reference/dm_limits_and_restrictions#on-request-capacity), and for the request process, see [Request additional capacity](/cdf/dm/records/guides/choose_and_size_streams#request-additional-capacity). Once you create a stream with a specific template, you cannot change that template. If you need different template settings, you must delete all records from the stream, delete the stream, then create a new stream with the correct template. #### Stream naming rules and limits Stream `externalId` values must start with a lowercase letter, contain only lowercase letters, digits, hyphens, and underscores, and be at most 100 characters long. The value must match this pattern: `^[a-z]([a-z0-9_-]{0,98}[a-z0-9])?$`. The number of active streams per project is limited. An **active stream** is any stream that exists and has not been deleted. Soft-deleted streams (streams that have been deleted but are still within their template's recovery window) do not count toward this limit. For current limits, see [Stream limits per project](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-limits-per-project). Each stream template defines specific limits for records, storage, and throughput. For the limit values of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). If you're building applications or services that use Records, implement the [recommended approaches](/api-reference/concepts/20230101/rate-limits) for managing concurrency and rate limits to avoid hitting these limits. #### Query time range limits Immutable streams require a `lastUpdatedTime` range on every `filter` and `aggregate` query. The `maxFilteringInterval` setting on each stream template defines the maximum span between the `gt` (start) and `lt` (end) timestamps in a **single request**. The limit follows from how the two stream types store data. Immutable streams write records into time-based partitions that roll over as data arrives, with every property indexed within each partition. The `lastUpdatedTime` range bounds how many partitions one request touches, so query performance stays stable as a stream grows to billions of records. Mutable streams keep one continuously updated index instead, which is why they don't need the range and why their capacity limits are much lower. For example, the `BasicArchive` template has a `maxFilteringInterval` of 365 days. This means each request can cover at most a 365-day window, but this window can be **anywhere** in the stream's history, not just relative to the current date. If the difference between `gt` and `lt` exceeds the interval, the API returns a validation error. Since `BasicArchive` has unlimited data retention, all historical data remains accessible. To query data spanning more than 365 days, split your requests into adjacent time windows that each stay within the limit. The following table shows an example for a multi-year query. | Request | `gte` | `lt` | | - | - | - | | 1 | `2022-01-01T00:00:00Z` | `2023-01-01T00:00:00Z` | | 2 | `2023-01-01T00:00:00Z` | `2024-01-01T00:00:00Z` | | 3 | `2024-01-01T00:00:00Z` | `2025-01-01T00:00:00Z` | | 4 | `2025-01-01T00:00:00Z` | *(now)* | You can check a stream's current `maxFilteringInterval` by retrieving the stream at `/api/v1/projects/{project}/streams/{streamId}` and inspecting `settings.limits.maxFilteringInterval`. The value is an ISO 8601 duration (for example, `P1Y` for one year or `P7D` for seven days). For mutable streams, `lastUpdatedTime` is optional, but using it improves query performance. #### Pagination and data retrieval The Records API provides three endpoints for consuming data, each with different pagination behavior: * **`filter`**: Returns up to 1,000 records in a single response. This endpoint does **not** support cursor-based pagination. It's designed for interactive queries where you need custom sorting and expect a bounded result set. If you need to retrieve more than 1,000 matching records, use the `sync` endpoint instead. * **`sync`**: The only endpoint that supports **cursor-based pagination**. It returns up to 1,000 records per page, and you iterate through results by passing the cursor from the previous response. Use this endpoint for batch processing, data exports, or any workflow that needs to process large volumes of records. The `sync` endpoint provides the same filtering capabilities as `filter`, but does not support custom sorting. * **`aggregate`**: Returns all results in a single response. Cursor-based pagination is **not** supported because aggregations compute over the entire matching dataset. The result size is inherently bounded by the aggregation structure, not by the number of individual records. There's no endpoint that retrieves a single record by identifier. To fetch one record, use `filter` with `equals` on `["space"]` and `["externalId"]`. On an immutable stream, the same identifier can match several records, one per version. #### Deleting streams Streams are **resource-intensive, long-lived entities** designed to persist for the lifetime of your project. Plan your stream strategy carefully and avoid patterns that involve repeatedly creating and deleting streams. When you delete a stream, it enters a **soft-deleted state** to protect against accidental data loss. Streams have no backup mechanism, so soft-delete is the only way to recover from accidental deletion. During this period: * The stream and its data are preserved but inaccessible (no ingestion or queries). * The stream doesn't count toward the active stream limit. * The stream's `externalId` is **reserved** and cannot be reused for a new stream until the soft-delete period expires. * You can recover the stream by contacting [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new). The duration of the soft-deleted state depends on the stream template. After this period, the stream and its data are permanently deleted, and the `externalId` becomes available for reuse. A single project can have a limited number of soft-deleted streams at any given time. To avoid hitting this limit, avoid creating and deleting streams frequently. We expect streams to be long-lived. The exception is streams created with one of the `test` templates. Deleting a stream can take a long time, depending on the stream settings and the volume of data stored. For limits on active and soft-deleted streams, see [Stream limits per project](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-limits-per-project). #### Data retention Some stream templates define a `dataDeletedAfter` retention period that controls how long records are kept before they are automatically removed. The retention period is fixed when you create the stream and can't be changed afterwards. Automatic data retention **only applies to immutable streams**. Mutable streams do not support the `dataDeletedAfter` setting. If you need to manage the lifecycle of records in a mutable stream, you must handle it yourself. For example, by deleting records directly or archiving finalized records to an immutable stream with a retention policy. You can check a stream's retention setting by retrieving the stream at `/api/v1/projects/{project}/streams/{streamId}` and inspecting `settings.lifecycle.dataDeletedAfter`. The value is an ISO 8601 duration (for example, `P7D` for seven days). If this field is absent, the stream has unlimited retention. Record deletion after the retention period is **not instantaneous**. Expired records are removed in large batches, and there may be a delay of up to several days between a record's expiration and its actual deletion. During this time, expired records remain queryable and continue to count toward the stream's storage and record count limits. Plan your stream capacity accordingly, accounting for this buffer when estimating total storage needs. ### Records Records are individual data objects that represent events, logs, or historical entries. Whether a record is immutable or mutable depends on the stream template you choose when creating the stream. An industrial knowledge graph describes relationships between entities using nodes and edges. Nodes can represent physical entities, such as equipment, or logical concepts, such as activities and process stages. However, when handling bulk data such as logs or historical records, storing each individual record as a node increases relational complexity and degrades query and retrieval performance. The following diagram represents this anti-pattern you should avoid. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: { "theme":"base", "themeVariables": { "lineColor":"#b8bec6", "primaryColor":"#ffffff", "fontFamily":"Inter, system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans, Helvetica Neue, Arial, sans-serif", "fontSize":"14px" } }}%% graph LR %% ===== Node styles (colors match the PNGs closely) ===== classDef Enterprise fill:#e53935,stroke:#24292f,stroke-width:2px,color:#fff classDef Site fill:#2f6cf6,stroke:#24292f,stroke-width:2px,color:#fff classDef Area fill:#8e43e7,stroke:#24292f,stroke-width:2px,color:#fff classDef Line fill:#0fa678,stroke:#24292f,stroke-width:2px,color:#fff classDef Equip fill:#f39c12,stroke:#24292f,stroke-width:2px,color:#fff classDef Record fill:#566370,stroke:#24292f,stroke-width:2px,color:#fff %% Thicker neutral edges by default linkStyle default stroke:#b8bec6,stroke-width:3px; %% ===== Layout (Left-to-right tree) ===== E((Enterprise)):::Enterprise SP((Site/Plant)):::Site A1((Area 1)):::Area A2((Area 2)):::Area A3((Area 3)):::Area L1((Line 1)):::Line L2((Line 2)):::Line L3((Line 3)):::Line EM1((Equipment
modules)):::Equip EM2((Equipment
modules)):::Equip EM3((Equipment
modules)):::Equip R1((Historical
record)):::Record R2((Historical
record)):::Record R3((Historical
record)):::Record R4((Historical
record)):::Record %% ===== Structure ===== E --> SP SP --> A1 SP --> A2 SP --> A3 A1 --> L1 A2 --> L2 A3 --> L3 L1 --> EM1 L2 --> EM2 L3 --> EM3 %% Dotted fan-out "Direct relations" from the middle equipment EM2 -. "Direct relations" .-> R1 EM2 -.-> R2 EM2 -.-> R3 EM2 -.-> R4 ``` Use the Records service to avoid these performance penalties for high-volume data. Records, together with [streams](#streams), let you store high-volume structured data in bulk, improving both the performance and scalability of your CDF-based solutions. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: { "theme":"base", "themeVariables": { "lineColor":"#b8bec6", "primaryColor":"#ffffff", "fontFamily":"Inter, system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans, Helvetica Neue, Arial, sans-serif", "fontSize":"14px" } }}%% graph LR %% ===== Node styles (colors match the PNGs closely) ===== classDef Enterprise fill:#e53935,stroke:#24292f,stroke-width:2px,color:#fff classDef Site fill:#2f6cf6,stroke:#24292f,stroke-width:2px,color:#fff classDef Area fill:#8e43e7,stroke:#24292f,stroke-width:2px,color:#fff classDef Line fill:#0fa678,stroke:#24292f,stroke-width:2px,color:#fff classDef Equip fill:#f39c12,stroke:#24292f,stroke-width:2px,color:#fff classDef Record fill:#566370,stroke:#24292f,stroke-width:2px,color:#fff classDef Transparent fill:transparent,stroke:none,stroke-width:0px %% Thicker neutral edges by default linkStyle default stroke:#b8bec6,stroke-width:3px; %% ===== Layout (Left-to-right tree) ===== E((Enterprise)):::Enterprise SP((Site/Plant)):::Site A1((Area 1)):::Area A2((Area 2)):::Area A3((Area 3)):::Area L1((Line 1)):::Line L2((Line 2)):::Line L3((Line 3)):::Line EM1((Equipment
modules)):::Equip EM2((Equipment
modules)):::Equip EM3((Equipment
modules)):::Equip %% ===== Filter View card (table) ===== FV@{ img: "table.svg" } class FV Transparent %% ===== Structure ===== E --> SP SP --> A1 SP --> A2 SP --> A3 A1 --> L1 A2 --> L2 A3 --> L3 L1 --> EM1 L2 --> EM2 L3 --> EM3 %% ---- Layout helper (keeps FV at the far right) ---- %% This solid edge is ONLY to pin FV to the right; we hide it with linkStyle below. EM2 --> FV %% ---- Visible dotted relation with left-pointing arrow ---- FV -. "Direct relation" .-> EM2 %% Make the layout helper edge invisible (this is the 11th edge: index 10) linkStyle 10 stroke:#000,stroke-width:0,opacity:0; ``` Immutability is a key design feature for records that guarantees historical records cannot be altered, while also delivering cost-effective support for massive storage volumes. Although records support mutability through mutable stream templates, updating records comes at a significant processing and ingestion cost compared to data modeling instances. Use mutable streams as a **transitional stage** for data that needs updates during its lifecycle, then archive finalized records to an immutable stream for permanent storage. #### Updating records in mutable streams Records do **not** support partial updates. When you upsert a record in a mutable stream, you must provide the **complete state** of the record, including all properties for the container. The upsert operation replaces the entire record, any properties you omit are not preserved from the previous version. This means that all **non-nullable properties must be included** in every upsert request, even if you only want to change one field. Omitting a non-nullable property causes a validation error. The recommended workflow for updating a record is: 1. **Read** the existing record using the `filter` or `sync` endpoint. 2. **Merge** your changes into the full property set. 3. **Upsert** the complete record back to the stream. This differs from [data modeling instances](/cdf/dm/dm_concepts/dm_spaces_instances#instance), which support partial updates where you only need to send the properties you want to change. For records, every upsert is a full replacement. #### Records vs. nodes | | Data modeling nodes | Records | | - | - | - | | Storage | Entities in the industrial knowledge graph | Data stored in streams as large batches | | Identification | Instances must have a unique external ID per node in the space. | See [Identifiers for records](#identifiers-for-records) | | Mutability | Mutable by default, immutable by configuration | Depends on the stream template applied when creating the stream | | Data volumes | Millions of nodes with low growth (once the initial graph is defined) | Billions of records per year continuously | | Structure | Structured using containers, option to use JSON for semi-structured data | Structured using containers, option to use JSON for semi-structured data | | Relationships | Many-to-many in a mesh, defined by edges and direct relations between instances | Connected to data modeling instances via direct relations in the records | #### Identifiers for records In data modeling, you identify nodes using a combination of the space ID and the mandatory node external ID. The external ID must be unique within the space it's scoped to, but you can reuse the same external ID across different spaces. Records also use external IDs. Like data modeling nodes, a record's external ID belongs to a space and is stored in a stream that can include records from multiple spaces. For records, the stream type determines the uniqueness constraints: * **Mutable streams**: the service enforces uniqueness for each combination of external ID, space ID, and stream. When you update a record with the same external ID/space/stream combination, it updates the existing record rather than creating a new one. * **Immutable streams**: the service does not enforce uniqueness. An immutable stream can contain multiple records with the same stream/space/external ID combination. This is useful for storing the full history of a record over time. You can use filtering capabilities to retrieve these records in bulk. In a single write request to a mutable stream, all combinations of `space` + `externalId` must be unique. You cannot create and update a record with the same space and external ID combination in the same POST request to `/streams/{streamId}/records`. ### Spaces Records use [data modeling spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space) for access control and organization. You must define a space before you can ingest records into it. Records can share spaces with data modeling instances, be stored in dedicated spaces, or use multiple shared spaces depending on your access control requirements. The following diagram illustrates three common space organization patterns: shared spaces where instances and records coexist, independent spaces for separate organization, and multiple shared spaces where records can belong to multiple spaces simultaneously. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: { "theme": "base", "themeVariables": { "lineColor": "#b8bec6", "primaryColor": "#ffffff", "tertiaryColor": "#f6f8fa", "fontFamily": "Inter, system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans, Helvetica Neue, Arial, sans-serif", "fontSize": "14px" } }}%% graph TB classDef SpaceClass fill:#0fa678,stroke:#24292f,stroke-width:2px,color:#fff classDef InstanceClass fill:#8e43e7,stroke:#24292f,stroke-width:2px,color:#fff classDef RecordClass fill:#566370,stroke:#24292f,stroke-width:2px,color:#fff linkStyle default stroke:#b8bec6,stroke-width:3px; %% ===== Shared space ===== subgraph "Shared space" Space5([Space 5]):::SpaceClass Instance1[("DM instances
Asset")]:::InstanceClass Instance2[("DM instances
Equipment")]:::InstanceClass Record1[["Records
Alarm Logs"]]:::RecordClass Record2[["Records
Event Data"]]:::RecordClass Instance1 --> Space5 Instance2 --> Space5 Record1 --> Space5 Record2 --> Space5 end %% ===== Independent spaces ===== subgraph "Independent spaces" Space3([Space 3
Instances only]):::SpaceClass Space4([Space 4
Records only]):::SpaceClass Instance3[("DM instances
Sensor")]:::InstanceClass Record3[["Records
Sensor Logs"]]:::RecordClass Instance3 --> Space3 Record3 --> Space4 end %% ===== Multiple shared spaces ===== subgraph "Multiple shared spaces" Space1([Space 1]):::SpaceClass Space2([Space 2]):::SpaceClass Instance4[("DM instances
Asset")]:::InstanceClass Record4[["Records
Alarm Logs"]]:::RecordClass Record5[["Records
Event Data"]]:::RecordClass Instance4 --> Space1 Instance4 --> Space2 Record4 --> Space1 Record4 --> Space2 Record5 --> Space2 end ``` Currently, you can delete a space that contains records. In an upcoming release, deleting a space with records will be prevented. You'll need to delete all records from the space before you can delete the space itself, making this consistent with data modeling behavior. If you delete a space and recreate it with the same external ID, the records will still be associated with the original space ID and remain accessible in queries. ### Record containers Record containers are [data modeling containers](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) that define the schema for records. You can **only** ingest records into containers with [`usedFor`](/cdf/dm/dm_concepts/dm_containers_views_datamodels#which-types-of-instances-can-you-use-a-container-for) set to `record`. Containers designated for records (`usedFor: record`) support significantly more properties than standard containers used for nodes and edges. See [Limits and restrictions](/cdf/dm/dm_reference/dm_limits_and_restrictions) for specific property limits. Deleting a container makes all records using that container **permanently inaccessible**, even if you recreate a container with the same external ID and schema. Plan your container schema carefully before ingesting records, as you cannot recover access to records after deleting their container. Not all container capabilities apply to records. When designing containers for records, review the following tables to understand which features are supported and which are not supported. #### Constraints and indexes Constraints and indexes are not available for record containers. | Feature | Data modeling instances | Records | | - | - | - | | **Constraints** (uniqueness, requires) | Supported | Not supported | | **Indexes** (BTree, inverted) | Supported | Not supported | #### Property settings and types Some individual property settings are not supported for record containers. | Feature | Data modeling instances | Records | | - | - | - | | **`immutable`** | Enforced. Updates to the property are rejected after initial write. | Not supported. Immutability is a stream-level concept only. | | **`autoIncrement`** | Supported. The system auto-generates incrementing values. | Not supported. Omitting a non-nullable auto-increment property causes ingestion failure. | | **`constraintState`** | Supported. Tracks validity of constraints on a property. | Not applicable. Records don't support constraints. | | **`enum` type** | Supported | Supported | | **`timeseries`, `file`, `sequence` types** | Supported | Not supported | | **`collation`** (text sorting rules) | Supported. Controls sort order using ICU collation rules. | Not supported | #### Direct relations Direct relation properties are supported in record containers, but with reduced validation and no auto-creation behavior. | Feature | Data modeling instances | Records | | - | - | - | | **Target validation** | Enforced. Validates both target space and instance existence. | Partially enforced. Validates target space existence only. | | **Auto-creation** | Supported. Auto-creates target nodes when `autoCreateDirectRelations` is set. | Not supported | | **`container` constraint** | Enforced. Validates the target matches the specified container type. | Not supported | ### Linking records to the knowledge graph You can link records to the knowledge graph by defining **direct relation properties** in your record container schema. These properties enable you to contextualize records by storing references to data modeling instances using their space and external ID. For instance, you can link sensor logs to specific wells or alarm records to particular assets. To link sensor logs to a `Well` in your data model: 1. **Define a direct relation property** in your record container schema, such as `well` of type `direct relation`. 2. **Assign the relationship** when ingesting records by providing the space and external ID of the target instance. 3. **Query and filter efficiently** using this property to retrieve all records associated with a specific instance. This approach enables the Records service to filter data efficiently without scanning entire streams. For example, you can run queries like "retrieve all sensor logs for well X" or "find all maintenance records for asset Y." **Validation behavior for direct relations** When you ingest records with direct relation properties, the Records service validates that the **target space exists** but does not verify that the referenced instance exists. This differs from standard data modeling behavior, where both space and instances are validated. You can ingest a record with a direct relation pointing to a non-existent instance, as long as the target space is valid. Design your ingestion pipelines to ensure referenced instances exist before creating records that link to them. When designing your container schema, include direct relation properties for instances you'll frequently use in filters and aggregations. #### Traversing between records and the graph Traversal betweeb Records and Data Modeling is a two-step query in either direction, and [record views](#record-views) make the link discoverable from both sides. * **Records to graph**: Filter records on the records side first (for example, critical alarms from the last hour), collect the distinct `{ "space", "externalId" }` values of the direct relation property that points to instances in the graph, and retrieve those instances from data modeling through the view the record view declares as `source`. Dangling references are simply absent from the result. * **Graph to records**: Filter instances on the data modeling side first (for example, all crude oil pumps), then query records with an `equals` filter on the direct relation property for a single instance, or an `in` filter with up to 100 `{ "space", "externalId" }` values in one request. To reference more than 100 instances, split them across several requests, or combine several `in` clauses with `or` within the filter limits. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sequenceDiagram participant App participant DM as Data Modeling API participant Rec as Records API App->>DM: Filter assets which are crude oil pumps DM-->>App: pump-001, pump-002 App->>Rec: Filter records where asset in [pump-001, pump-002] Rec-->>App: Alarm records for those pumps ``` ### Record views A record view is a named, versioned interface for reading and writing records. Record views follow the same separation of storage and consumption as [data modeling views](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views): containers define where properties are stored, while views define the property names and relationships that applications use. Use record views to keep applications independent of the physical record schema. A record view can: * Map properties from multiple record containers into one flat interface. * Rename container properties without changing stored data. * Associate the interface with a stream, so consumers don't need to know which stream to use. * Provide one source reference for ingest, upsert, filter, sync, and aggregate operations. * Add type information to direct relations by identifying the data modeling view used to read the target instances. The `streamId` field binds a record view to its backing stream: requests that use the view must target that stream, and clients can read the stream identifier from the view instead of hard-coding it. A record view can map multiple containers, but every mapped container must have `usedFor: record`. A regular data modeling view can't map record containers, and a record view can't map containers used for nodes or edges. For the full view schema and API operations, see [Views](/api-reference/concepts/20230101/views). Use the view identifier and view property identifiers in client code. The view resolves each exposed property to its underlying container property and preserves the stream association. You can therefore evolve storage behind a new view version without coupling consumers to container identifiers. Data models can't include record views yet, and records aren't returned by `/models/instances/query` or GraphQL. Like time series data points, records are only queryable through their own endpoints. Read and write records through the Records API endpoints, and traverse to the graph with the two-step pattern described in [Traversing between records and the graph](#traversing-between-records-and-the-graph). #### Implicit container filtering Reading through a view, with a view source or a view property path in a filter or aggregation, returns only records that have data in **all** containers the view maps (the view's `mappedContainers`). This matches data modeling views and is what lets one stream hold several record types: a view that maps a shared container and an alarm container returns alarms only, because other record types have no data in the alarm container. #### View-level filters A record view can carry a `filter`. Every read through the view, with `filter`, `sync`, and `aggregate`, applies it, combined with the implicit `hasData` container filtering and with the filter in the request. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: OpenAlarms version: v1 streamId: [alarms_live] filter: equals: property: [alarm_schema, alarm, isAcknowledged] value: false ``` A consumer who queries `OpenAlarms` with a priority filter in the request gets unacknowledged, high-priority alarms without mentioning acknowledgement at all. #### Implementing other record views Record views support `implements`, like data modeling views: a view that implements a base record view inherits the base view's property mappings and adds its own, so a family of views (a base `Alarm` view with specialized variants) doesn't remap the shared containers. Implemented views must be record views, and all mapped containers must be `usedFor: record`. #### Connect record views to data modeling views Record views make direct relations between records and data modeling instances discoverable from both sides of the relationship. On the record view, set the mapped direct relation property's `source` to the data modeling view that consumers should use to read the target instance. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sensor: container: type: container space: sensor_schema externalId: sensor_log_context containerPropertyIdentifier: sensor source: type: view space: opcua_schema externalId: Sensor version: v1 ``` On the data modeling view, define a reverse direct relation whose `source` and `through.source` reference the record view. Set `through.identifier` to the record view property that exposes the direct relation. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sensorLogs: connectionType: multi_reverse_direct_relation name: Sensor logs description: SensorLog records associated with this sensor source: type: view space: sensor_schema externalId: SensorLog version: v1 through: source: type: view space: sensor_schema externalId: SensorLog version: v1 identifier: sensor ``` This metadata anchors records to the knowledge graph. Consumers can traverse from a record to its related sensor through the data modeling view, or discover related records from a sensor through the reverse direct relation. The record data remains in the Records service, similar to how time series data points and file contents remain in their respective services while their resources are registered in the graph. Because the two views reference each other, create them in the same `POST /models/views` request. The request accepts data modeling views and record views together, so neither view has to exist before the other. To follow the relation at query time, see [Traversing between records and the graph](#traversing-between-records-and-the-graph). Record views can expose mapped direct relation properties, but they can't define edge or reverse direct relation connection properties. Define the reverse direct relation on the data modeling view instead. ## Capabilities Records and streams have their own [capabilities](/cdf/access/guides/capabilities#records) for access control. These capabilities are independent of each other and are not inherited from the data modeling service. However, because records rely on the data modeling container feature, you must have the `dataModels:READ` capability to read or write records in a stream. **Stream selection is not access-controlled.** Access control for records operates at the **space** level via `streamrecords:write` and `streamrecords:read` capabilities scoped to a space. However, there is currently no capability that restricts which stream a user can write into. Any user with write access to a space can choose any available stream when ingesting records. ## Data ingestion Records are ingested directly through the API, the SDKs, or the extractors that support streams. See [Get started with Records](/cdf/dm/records/guides/records_getting_started) for the supported ingestion paths. For scheduled or medium-volume sources, write from a [Cognite Function](/cdf/functions) or a [Data Workflow](/cdf/data_workflows) task that stays within the stream's rate limits. To move or enrich records between streams, for example to archive finalized records from a mutable stream to an immutable one, use a Data Workflow with the [record stream trigger](/cdf/data_workflows/triggers). The Records service operates with **near real-time consistency**. When you ingest or update records, there is typically a brief delay, **up to a few seconds**, between when the API returns a successful response and when the changes become visible in search results, filters, and aggregations. This delay occurs because the service periodically makes newly ingested data searchable, balancing performance for high-volume data ingestion with quick data availability. In most cases, new or updated records become searchable **within 1-2 seconds** of ingestion. Keep this near real-time consistency in mind when designing your application: * **Write-then-read scenarios**: If you ingest a record and immediately query for it, the record may not appear in the results yet. Consider implementing a brief retry mechanism or delay if your workflow depends on immediate read-after-write consistency. * **Immediate updates**: For use cases with low data volumes requiring immediate visibility of every update, consider using [data modeling instances](/cdf/dm/dm_concepts/dm_spaces_instances#instance) instead of records. ## Unit-aware queries Record containers support float properties with units from the CDF unit catalog. When you query records, you can request values in a different compatible unit without modifying the stored data. The `records/sync`, `records/filter`, and `records/aggregate` endpoints accept a top-level `targetUnits` parameter that converts both response values and filter inputs to your requested unit. For example, if your container stores `maxPressure` in bar, you can request results in pascal and write your range filters with pascal values — the service handles the conversion in both directions automatically. Set `includeTyping: true` to include a `typing` block in the response that shows the resolved unit for each property, which is useful for building dashboards that display unit labels dynamically. For the full units reference, code examples, and data normalization patterns, see [Units in CDF — Integration with records](/dev/concepts/resource_types/units/units#integration-with-records). ## Getting started To begin using records and streams effectively, start by identifying your high-volume data sources and the target structure you need. Then explore: * [Choose and size streams](/cdf/dm/records/guides/choose_and_size_streams). Decide whether records fit your workload, pick a default stream template, and request additional capacity when needed. * [Get started with Records](/cdf/dm/records/guides/records_getting_started) — Complete tutorial that walks you through creating schemas, setting up streams, ingesting records, querying, and building stream-to-stream pipelines. * [Aggregate records reference](/cdf/dm/records/guides/records_aggregations) — Use aggregations to compute statistics and analyze trends across records without retrieving individual items. * [Property paths](/cdf/dm/records/guides/records_aggregations#property-paths) — Reference properties through containers or record views in filters, sorts, and aggregations. * [Units in CDF — Integration with records](/dev/concepts/resource_types/units/units#integration-with-records) — Full reference and examples for unit-aware queries, filters, aggregations, and data normalization patterns. # About choosing and sizing streams Source: https://docs.cognite.com/cdf/dm/records/guides/choose_and_size_streams Decide whether records fit your workload, understand how stream limits shape sizing, pick a default template, and request additional capacity in records, storage, and retention in the industrial knowledge graph. Before you create production streams, use this article to decide whether records are the right CDF service for your data, which default stream template to use, and when to request capacity above the project defaults. For the limit values of each default template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). ## Evaluate whether to use records Records and streams provide high-performance storage for bulk structured data that lives next to the industrial knowledge graph. Typical examples include logs, events, notifications, archived history, and other append-mostly rows. Not every workload belongs in records. Use the table below to match your data to the right CDF service. | Your data | Use this CDF service | | - | - | | Numeric, string or state measurements indexed by time, one value per tag per timestamp (sensors, PI, OPC UA DA). | [Time series and datapoints](/api-reference/concepts/20230101/time-series-data) | | Entities in the physical or logical world with rich relationships and modest cardinality. | [Data modeling instances](/cdf/dm/dm_concepts/dm_spaces_instances#instance) | | Binary blobs like PDFs, P\&IDs, photos, 3D, and lab files in their original format. | [Files](/cdf/dm/dm_guides/dm_integrate_files) | | Append-mostly structured rows such as alarm events, OPC UA frames, batch phase records, MES logs, lab samples, audit trails, and well log curves by depth. | Records | ## Plan before you create streams Work through these planning questions before you start large-scale ingestion. Capture your answers in your project's architecture or capacity documentation. If you can't answer the lifecycle and volume questions, pause implementation and gather estimates first. * **Use case**. What data are you storing? How is it ingested and consumed? * **Lifecycle**. Are records updated or deleted after the first write? * **Retention**. Do you need data forever, or for a fixed number of years? The retention period is fixed when you create the stream. * **Volume**. How many records per day, month, and year? What is the total over your retention period? * **Streams**. Do you need one stream or several, for example by site, domain, or time window? * **Fit**. Does the workload fit default templates, or do you need a [capacity request](#request-additional-capacity)? ## Understand stream limits A stream template has a fixed set of limits applied when you create a stream. This section explains what each limit means for your design. For the current limit values of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). Request additional capacity **before** you create the stream. Per-stream limits are fixed at creation and can't be raised later. If a stream outgrows its template, migrate the data to a new stream with higher limits, then delete the old stream. ### Mutability **Immutable streams** (`ImmutableTestStream`, `BasicArchive`) don't allow you to update or delete individual records after ingestion. They support higher total record counts and storage volume. **Mutable streams** (`BasicLiveData`) allow update and delete. They're sized as a small, hot working set. The supported production pattern is one `BasicLiveData` stream per project for data that's still changing, plus one or more immutable archive streams for archival-state data. When a record reaches its archival state, write it to an immutable archive and delete it from the mutable stream. See [Updating records in mutable streams](/cdf/dm/records/concepts/records_and_streams#updating-records-in-mutable-streams). ### Max records and max data volume Together these cap the size of one stream. Typical industrial records are roughly 600 to 900 bytes (alarms, batch records, and event frames often fall in this range). **Record count** usually reaches the limit before **storage volume**. Plan against whichever binds first for your average record size. For example, a site ingesting 7.3 million alarm records per year (\~900 bytes each, \~6.1 GB/year) hits the 50 million record cap on a `BasicArchive` stream in about seven years. At that ingest rate, the 50 GB volume cap would take longer and record count would bind first. Every project starts with the default templates, and Cognite can enable more capacity on request. For the default caps, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits), and for the on-request range, see [On-request capacity](/cdf/dm/dm_reference/dm_limits_and_restrictions#on-request-capacity). ### Retention Retention (`dataDeletedAfter`) controls when records are automatically hard-deleted. The defaults differ by template: **archive streams** keep records until you delete the stream, **test streams** delete records after a short period, and **mutable streams** have no time-based retention — the record cap bounds mutable storage instead. For the default retention of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). Retention settings help you keep only relevant data within stream limits. If you require a custom retention policy, request it through your Cognite representative. ### Throughput Throughput is enforced per stream in 10-minute sliding windows. There are separate limits for records ingested, bytes ingested, bytes read, and updates and deletes (mutable streams only). For example, backfilling 45 GB at the `BasicArchive` ingest limit of 170 MB per 10 minutes takes about 44 hours of continuous ingestion (two days). Keep sustained load below 70% of the ceiling. Bursts and retries can throttle you if you run near the limit continuously. The limits are evaluated over the 10-minute window, but don't spend the whole allowance in a burst and then idle: a steady request rate gives better performance and predictable throttling than front-loading each window. ### Max range filter interval for `lastUpdatedTime` Immutable streams require a `lastUpdatedTime` range on every `filter` and `aggregate` request. Each template sets `maxFilteringInterval`, which is the largest window between `gt` and `lt` in one request. For example, the value is 365 days on `BasicArchive` and 7 days on `ImmutableTestStream`. To query a wider span, split your requests into adjacent time windows. See [Query time range limits](/cdf/dm/records/concepts/records_and_streams#query-time-range-limits). For mutable streams, `lastUpdatedTime` is optional on queries. Using it improves query performance. ### Max unique properties The max unique properties limit is the count of distinct property names with data across the entire stream, not per record. Every property that's ever been written through a [container](/cdf/dm/dm_concepts/dm_containers_views_datamodels) counts toward the limit. If you have three containers, each with 200 properties, the stream uses up to 600 unique properties. If you approach the limit, review your schema layout or group rarely queried fields in a JSON property. If you still exceed the limit, contact your Cognite representative to discuss your use case. The consumed property count does not decrease when you stop writing a property, and deleting a container doesn't release the slots it consumed. On immutable streams, you also can't delete individual records to free slots. There's no API today to report how many unique properties a stream has consumed. Track the properties defined on the [containers](/cdf/dm/dm_concepts/dm_containers_views_datamodels) that write into each stream as part of your capacity planning. ### Soft-delete retention When you delete a stream, it enters a soft-deleted state. You can request recovery through [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new) within a template-specific window. For the recovery window of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). Restoration is manual and typically takes about 2 weeks after you submit the request, so contact Support as soon as you notice the deletion. Don't rely on soft-delete as a backup strategy. Projects also limit how many soft-deleted streams you can have at once. See [Stream limits per project](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-limits-per-project). ### Active streams per project Each project limits the total number of active streams and the number of active streams per template. For the current values, see [Stream limits per project](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-limits-per-project). You can use different template types within the total active stream cap. To raise these limits, see [Request additional capacity](#request-additional-capacity). If you're considering several streams to work around per-stream limits, discuss the design with Cognite first. Multiple streams add cost and complexity, and you can't query across streams. In most cases, raising limits on one stream is a better fit than splitting the workload across several smaller streams. ### Sizing example: multi-site alarms A specialty chemical company consolidates DCS alarms from 10 plants: | Input | Value | | - | - | | Records per year | 7.3 M/plant x 10 = 73 M/year | | Average record size | \~900 B, or \~66 GB/year | | Retention | 5 years | | Backfill | 5 years (365 M records, \~330 GB) in 30 days, then steady state | Fit against defaults: | Check | Result | | - | - | | Fits `BasicArchive` alone? | No. 365 M exceeds the 50 M record cap and 330 GB exceeds the 50 GB volume cap. | | Steady-state ingest | \~1,390 records per 10 minutes, well below the 170,000 limit. | | Backfill ingest | \~76 MB per 10 minutes, under half of the 170 MB limit. | **Streams to provision** When you request additional capacity, ask for record count, volume, and retention above your projection so the stream has room to grow without another capacity request. | Stream | Configuration | Fits defaults? | | - | - | - | | All-plant alarm archive | Immutable, \~365 M records, \~330 GB, 5-year retention | No. The volume exceeds `BasicArchive` defaults. [Request additional capacity](#request-additional-capacity) with headroom above the 5-year projection. | | Active alarm acknowledgements | Mutable, default `BasicLiveData` capacity | Yes. No capacity request needed. | | Single-plant batch pilot | Immutable, \~10 M records, \~10 GB, 5-year retention | Volume fits `BasicArchive`, but the 5-year retention is non-default. [Request additional capacity](#request-additional-capacity) to set the retention with headroom. | ## Choose a default template CDF projects ship with three default stream templates: | Template | Use for | | - | - | | `BasicArchive` | Append-only history at single-facility scale. | | `BasicLiveData` | The production mutable working set for records that still change. | | `ImmutableTestStream` | Short experiments only. Never use for production. | For exact limits per template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits) in Limits and restrictions. Use the table below to pick the right template after you work through the [planning questions](#plan-before-you-create-streams). | If… | Choose | | - | - | | Records are never updated or deleted after ingestion in production. | `BasicArchive` | | Only a small subset changes until each record reaches its archival state, then history is append-only. | `BasicLiveData` for the working set plus `BasicArchive` for the archive, with an [archival flow](/cdf/dm/records/guides/records_getting_started). | | Your pipeline upserts or updates records from RAW, Functions, or extractors. | `BasicLiveData`, or change the pipeline to append-only before using `BasicArchive`. | | You need more than 5 million records in one mutable stream. | [Request additional capacity](#request-additional-capacity) before go-live. | | You need more than 50 million records in one immutable stream, or a non-default retention. | [Request additional capacity](#request-additional-capacity) before go-live. | | You're running a short experiment, benchmark, or schema iteration. | `ImmutableTestStream` | Immutable streams don't support updating or deleting individual records. If you switch from `BasicLiveData` to `BasicArchive` to gain capacity, pipelines that upsert into the stream fail until you change them to append-only ingestion. ## Request additional capacity Submit a capacity request if your sizing exceeds the [default per-stream limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits), you need more [active streams than the project default](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-limits-per-project), or you need a retention period the defaults don't provide. Describe what you need in record count, storage volume in GB, and retention per stream. For the upper range we can provision on request, see [On-request capacity](/cdf/dm/dm_reference/dm_limits_and_restrictions#on-request-capacity). If you chose the wrong template or outgrow a stream, migrate the data to another stream with sufficient capacity and then delete the old stream. See [Understand stream limits](#understand-stream-limits) for why limits can't be raised after creation. Migration is a client-side process: read the records from the old stream with `sync` and write them to the new one. To submit a request: 1. If you have a Cognite Customer Success or Solution Architect contact, send the request to them. 2. Otherwise, open a [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new) request and mention **Records / Streams custom capacity**. Routine requests typically take 1 to 2 weeks. Requests that need a new limit configuration may take longer. ## Next steps * [About records and streams](/cdf/dm/records/concepts/records_and_streams) for core concepts and query behavior. * [Get started with Records](/cdf/dm/records/guides/records_getting_started) for a hands-on alarm pipeline tutorial. * [Limits and restrictions](/cdf/dm/dm_reference/dm_limits_and_restrictions#records-and-streams) for stream counts, template limits, and on-request capacity. # Records aggregation reference Source: https://docs.cognite.com/cdf/dm/records/guides/records_aggregations Use aggregations to compute statistics, group data into buckets, and analyze trends across your records without retrieving individual items. Aggregations let you compute statistics, group data into buckets, and analyze trends across large volumes of records without retrieving individual items. Instead of fetching thousands of records and processing them client-side, a single aggregation request can return counts, averages, distributions, time-series breakdowns, and more, directly from the Records service. Common use cases include: * **Dashboards and reporting**: Power real-time dashboards with summary statistics, distributions, and time-histogram charts from a single API call. * **Trend analysis**: Track how metrics change over time using time-based histograms and moving window functions. * **Distribution analysis**: Understand how values are spread across categories, severity levels, or numeric ranges. * **Threshold monitoring**: Count records that fall into specific value ranges to detect anomalies or trigger alerts. Records supports three categories of aggregations: | Category | Aggregates | Description | | - | - | - | | **Metric** | `count`, `avg`, `sum`, `min`, `max` | Compute a single numeric value from a set of records | | **Bucket** | `uniqueValues`, `numberHistogram`, `timeHistogram`, `filters` | Group records into buckets based on property values, numeric ranges, time intervals, or filter conditions | | **Pipeline** | `movingFunction` | Compute derived values from the output of bucket aggregations, such as moving averages over time | ## Example schema The examples on this page use records stored in a stream with the following container schema. The container `equipment_events` in space `factory-data` tracks equipment monitoring events across an industrial facility. | Property | Type | Description | | - | - | - | | `equipment_id` | text | Equipment identifier (e.g., `PUMP-001`, `MOTOR-042`) | | `location` | text | Facility location (e.g., `Building-A`, `Building-B`) | | `temperature` | float64 | Measured temperature in degrees Celsius | | `severity` | text | Event severity level (`LOW`, `MEDIUM`, `HIGH`, `CRITICAL`) | | `priority` | int64 | Priority score from 1 (lowest) to 100 (highest) | | `is_critical` | boolean | Whether the event requires immediate attention | | `recorded_at` | timestamp | When the event was recorded | To learn how to create containers, set up streams, and ingest records, see [Get started with Records](/cdf/dm/records/guides/records_getting_started). ## Request structure Send aggregation requests to the aggregate endpoint: ``` POST /api/v1/projects/{project}/streams/{streamId}/records/aggregate ``` Every request requires an `aggregates` object that maps your chosen names to aggregate definitions. You can optionally include a `filter` to narrow the dataset and a `lastUpdatedTime` range. ```json Example request structure theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [ { "type": "container", "space": "factory-data", "externalId": "equipment_events" } ] }, "aggregates": { "my_aggregate_name": { "": { ... } } } } ``` **Time-based filtering with lastUpdatedTime** The `lastUpdatedTime` filter is **mandatory** for aggregate queries on immutable streams. Immutable streams are optimized for high-volume, append-only data and are internally organized by time. The service uses `lastUpdatedTime` to efficiently target only the relevant data partitions, avoiding costly full-stream scans across potentially billions of records. Without this constraint, aggregate queries on large immutable streams would be prohibitively slow. The filter defines the time range for the aggregation: * **`gt` (greater than)** / **`gte` (greater than or equal to)**: The start of the time range (required for immutable streams). * **`lt` (less than)** / **`lte` (less than or equal to)**: The end of the time range (optional, defaults to current time). The maximum time range you can query **in a single request** is determined by the stream's `maxFilteringInterval` setting (an ISO 8601 duration, for example `P1Y` for one year). This limit is defined by the [stream template](/cdf/dm/records/concepts/records_and_streams#stream-templates) selected when the stream was created, and it applies to the span between `gt` and `lt`, not to how far back in time you can reach — you can query any historical period as long as each request stays within the interval. If the difference between `gt` and `lt` exceeds this interval, the API returns a validation error. To query data spanning more than this interval, split your requests into adjacent time windows. See [Query time range limits](/cdf/dm/records/concepts/records_and_streams#query-time-range-limits) for details and examples. For mutable streams, `lastUpdatedTime` is optional, but using it improves query performance. ### Property paths Aggregations reference container properties using an array of three strings: the space, the container external ID, and the property name. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "property": ["factory-data", "equipment_events", "temperature"] ``` Some aggregations also support top-level record properties with a single-element array: * `["createdTime"]` — when the record was created * `["lastUpdatedTime"]` — when the record was last updated * `["space"]` — the space the record belongs to (only for `uniqueValues`) Aggregations can also reference properties through a [record view](/cdf/dm/records/concepts/records_and_streams#property-paths-through-views). Use the view's external ID and version as the second element and the view's property identifier as the third: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "property": ["factory-data", "EquipmentEvent/v1", "temperature"] ``` A view path applies the view's context: only records with data in all containers the view maps are aggregated, and the view-level filter, if the view defines one, is applied as well. ### Naming rules You choose the name for each aggregate in the `aggregates` map. Names must be 1-255 characters and cannot contain `[`, `]`, or `>`, since these characters are reserved for referencing aggregate paths in pipeline aggregations. **Cognite Python SDK imports** Aggregate request builders live in `cognite.client.data_classes.data_modeling.aggregates`, while `TimeRange` lives in `cognite.client.data_classes.data_modeling.records`. ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.data_modeling.aggregates import Average, Count, TimeHistogram from cognite.client.data_classes.data_modeling.records import TimeRange ``` The aggregates module is not re-exported from `cognite.client.data_classes.data_modeling`. That namespace already defines `Average`, `Count`, `Min`, `Max`, and `Sum` classes for the `instances.aggregate` endpoint. If you use both endpoints in the same file, import the records aggregates module under an alias: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.data_modeling import aggregates as aggs aggs.Average(["factory-data", "equipment_events", "temperature"]) ``` ## Metric aggregates Metric aggregates compute a single numeric value from all records that match your filter. You can combine multiple metric aggregates in a single request, similar to running `SELECT COUNT(*), AVG(temperature), SUM(priority), MIN(temperature), MAX(temperature) FROM equipment_events WHERE ...` in SQL. **Null value handling** For all metric aggregates except `count` without a property, records with a `null` or missing value for the specified property are excluded from the calculation. They do not affect the result. For example, `avg` computes the average only over records that have a non-null value, so 10 records where 3 have null temperatures would compute the average from the remaining 7 records. The `count` aggregate without a `property` counts all matching records regardless of null values. With a `property` specified, `count` only counts records where that property has a non-null value. ### count Returns the number of records. Without a `property`, it counts all matching records. With a `property`, it counts only records that have a non-null value for that property. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "total_events": { "count": {} }, "events_with_temperature": { "count": { "property": ["factory-data", "equipment_events", "temperature"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.aggregates import Count from cognite.client.data_classes.data_modeling.records import TimeRange client = CogniteClient() space_id = "factory-data" container_id = "equipment_events" stream_id = "your-stream-id" result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "total_events": Count(), "events_with_temperature": Count(property=[space_id, container_id, "temperature"]), }, ) print(f"Total events: {result['total_events'].value}") print(f"Events with temperature: {result['events_with_temperature'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'factory-data'; const containerId = 'equipment_events'; const streamId = 'your-stream-id'; const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { total_events: { count: {} }, events_with_temperature: { count: { property: [spaceId, containerId, 'temperature'] }, }, }, }); console.log(`Total events: ${result.total_events.count}`); console.log(`Events with temperature: ${result.events_with_temperature.count}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "total_events": { "count": 1250 }, "events_with_temperature": { "count": 1183 } } } ``` ### avg Computes the arithmetic mean of a numeric property across all matching records. Records with null values for the specified property are excluded from the calculation. **Supported property types**: `int32`, `int64`, `float32`, `float64` ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "avg_temperature": { "avg": { "property": ["factory-data", "equipment_events", "temperature"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import Average result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "avg_temperature": Average(property=[space_id, container_id, "temperature"]), }, ) print(f"Average temperature: {result['avg_temperature'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { avg_temperature: { avg: { property: [spaceId, containerId, 'temperature'] }, }, }, }); console.log(`Average temperature: ${result.avg_temperature.avg}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "avg_temperature": { "avg": 72.45 } } } ``` ### sum Computes the total sum of values for a numeric property. Useful for calculating totals like cumulative priority scores or aggregated measurements. **Supported property types**: `int32`, `int64`, `float32`, `float64` ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "total_priority": { "sum": { "property": ["factory-data", "equipment_events", "priority"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import Sum result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "total_priority": Sum(property=[space_id, container_id, "priority"]), }, ) print(f"Total priority: {result['total_priority'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { total_priority: { sum: { property: [spaceId, containerId, 'priority'] }, }, }, }); console.log(`Total priority: ${result.total_priority.sum}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "total_priority": { "sum": 58420 } } } ``` ### min Returns the lowest value for a property. Works with numeric and timestamp properties, including the top-level `createdTime` and `lastUpdatedTime` fields. **Supported property types**: `int32`, `int64`, `float32`, `float64`, `timestamp`, `date` ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "coldest_reading": { "min": { "property": ["factory-data", "equipment_events", "temperature"] } }, "earliest_event": { "min": { "property": ["lastUpdatedTime"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import Min result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "coldest_reading": Min(property=[space_id, container_id, "temperature"]), "earliest_event": Min(property=["lastUpdatedTime"]), }, ) print(f"Coldest reading: {result['coldest_reading'].value}") print(f"Earliest event: {result['earliest_event'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { coldest_reading: { min: { property: [spaceId, containerId, 'temperature'] }, }, earliest_event: { min: { property: ['lastUpdatedTime'] }, }, }, }); console.log(`Coldest reading: ${result.coldest_reading.min}`); console.log(`Earliest event: ${result.earliest_event.min}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "coldest_reading": { "min": 18.3 }, "earliest_event": { "min": "2025-10-01T08:12:33.000Z" } } } ``` ### max Returns the highest value for a property. Works with the same property types as `min`. **Supported property types**: `int32`, `int64`, `float32`, `float64`, `timestamp`, `date` ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "hottest_reading": { "max": { "property": ["factory-data", "equipment_events", "temperature"] } }, "latest_event": { "max": { "property": ["lastUpdatedTime"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import Max result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "hottest_reading": Max(property=[space_id, container_id, "temperature"]), "latest_event": Max(property=["lastUpdatedTime"]), }, ) print(f"Hottest reading: {result['hottest_reading'].value}") print(f"Latest event: {result['latest_event'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { hottest_reading: { max: { property: [spaceId, containerId, 'temperature'] }, }, latest_event: { max: { property: ['lastUpdatedTime'] }, }, }, }); console.log(`Hottest reading: ${result.hottest_reading.max}`); console.log(`Latest event: ${result.latest_event.max}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "hottest_reading": { "max": 142.7 }, "latest_event": { "max": "2025-10-24T16:45:12.000Z" } } } ``` ### Combined metrics example You can request multiple metric aggregates in a single call to get a complete statistical summary without making separate requests. This example also demonstrates how to combine `hasData` with a `range` condition using `and` to compute metrics only for records where the temperature exceeds 85.5 degrees Celsius. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "and": [ { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, { "range": { "property": ["factory-data", "equipment_events", "temperature"], "gt": 85.5 } } ] }, "aggregates": { "total_events": { "count": {} }, "avg_temp": { "avg": { "property": ["factory-data", "equipment_events", "temperature"] } }, "min_temp": { "min": { "property": ["factory-data", "equipment_events", "temperature"] } }, "max_temp": { "max": { "property": ["factory-data", "equipment_events", "temperature"] } }, "total_priority": { "sum": { "property": ["factory-data", "equipment_events", "priority"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples temperature = [space_id, container_id, "temperature"] priority = [space_id, container_id, "priority"] result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={ "and": [ {"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, {"range": {"property": temperature, "gt": 85.5}}, ] }, aggregates={ "total_events": Count(), "avg_temp": Average(property=temperature), "min_temp": Min(property=temperature), "max_temp": Max(property=temperature), "total_priority": Sum(property=priority), }, ) print(f"Total events: {result['total_events'].value}") print(f"Avg temp: {result['avg_temp'].value}") print(f"Min temp: {result['min_temp'].value}") print(f"Max temp: {result['max_temp'].value}") print(f"Total priority: {result['total_priority'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { and: [ { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, { range: { property: [spaceId, containerId, 'temperature'], gt: 85.5, }, }, ], }, aggregates: { total_events: { count: {} }, avg_temp: { avg: { property: [spaceId, containerId, 'temperature'] } }, min_temp: { min: { property: [spaceId, containerId, 'temperature'] } }, max_temp: { max: { property: [spaceId, containerId, 'temperature'] } }, total_priority: { sum: { property: [spaceId, containerId, 'priority'] } }, }, }); console.log(`Total events: ${result.total_events.count}`); console.log(`Avg temp: ${result.avg_temp.avg}`); console.log(`Min temp: ${result.min_temp.min}`); console.log(`Max temp: ${result.max_temp.max}`); console.log(`Total priority: ${result.total_priority.sum}`); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "total_events": { "count": 312 }, "avg_temp": { "avg": 104.8 }, "min_temp": { "min": 85.9 }, "max_temp": { "max": 142.7 }, "total_priority": { "sum": 15230 } } } ``` The `and` filter narrows the dataset to only records from the `equipment_events` container where `temperature` is above 85.5 degrees Celsius, so all returned metrics reflect that subset. You can use any [filter expression](/cdf/dm/dm_concepts/dm_querying) here, including `or`, `not`, `equals`, `prefix`, and nested combinations. ## Bucket aggregates Bucket aggregates group records into categories (called buckets) based on property values, numeric ranges, time intervals, or filter conditions. Each bucket includes a count of matching records and can contain nested sub-aggregates that compute additional metrics per bucket. Bucket aggregates with nested sub-aggregates work like `GROUP BY` with aggregate functions in SQL or pandas/Polars. A `uniqueValues` aggregate on `severity` with a nested `avg` on `temperature` is conceptually equivalent to: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT severity, COUNT(*) AS count, AVG(temperature) AS avg_temp FROM equipment_events GROUP BY severity ``` The key difference is that Records aggregates support multiple levels of nesting in a single request, similar to using `MultiIndex` grouping in pandas or chaining `.group_by()` operations in Polars. ### uniqueValues Groups records by the distinct values of a property and returns one bucket per unique value, ordered by count (highest first). This is the equivalent of `GROUP BY` in SQL, `.groupby()` in pandas, or `.group_by()` in Polars, useful for understanding distributions, for example, how many events exist for each severity level or which equipment generates the most events. #### Parameters | Parameter | Type | Required | Description | | - | - | - | - | | `property` | array | Yes | The property to group by. Supports text, numeric, boolean, and enum properties, and the top-level `["space"]` path. | | `size` | integer | No | Maximum number of buckets to return. Range: 1-2000. Default: 10. | | `aggregates` | object | No | Nested sub-aggregates to compute within each bucket. | **Ordering and size behavior** Buckets are returned in descending order by record count (most common values first). When your data contains more unique values than the requested `size`, the response returns only the top buckets by count. The remaining values are not included in the response. If you need to see all unique values, increase `size` up to the maximum of 2000. **Approximate counts** Counts may be approximate when the dataset is very large. Because records are distributed across multiple partitions for performance, each partition independently identifies its top values and these are merged to produce the final result. This means a value that is moderately common across many partitions might not appear in every partition's top list, leading to a slight undercount. For most use cases, the approximation is negligible and does not affect the relative ranking of top values. **Missing values** Records that do not have the specified property (or have a `null` value) are excluded from the aggregation. They are not counted and do not produce a bucket. Only records with a non-null value for the property contribute to the unique value buckets. #### Example: severity distribution with nested average ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "by_severity": { "uniqueValues": { "property": ["factory-data", "equipment_events", "severity"], "size": 10, "aggregates": { "avg_temp": { "avg": { "property": ["factory-data", "equipment_events", "temperature"] } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import UniqueValues result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "by_severity": UniqueValues( property=[space_id, container_id, "severity"], size=10, aggregates={ "avg_temp": Average(property=[space_id, container_id, "temperature"]), }, ), }, ) for bucket in result["by_severity"].buckets: print(f"{bucket.value}: {bucket.count} events, avg temp {bucket.aggregates['avg_temp'].value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { by_severity: { uniqueValues: { property: [spaceId, containerId, 'severity'], size: 10, aggregates: { avg_temp: { avg: { property: [spaceId, containerId, 'temperature'] }, }, }, }, }, }, }); const buckets = result.by_severity.uniqueValueBuckets; buckets.forEach((bucket) => { console.log(`${bucket.value}: ${bucket.count} events, avg temp ${bucket.aggregates.avg_temp.avg}`); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "by_severity": { "uniqueValueBuckets": [ { "value": "LOW", "count": 523, "aggregates": { "avg_temp": { "avg": 65.2 } } }, { "value": "MEDIUM", "count": 412, "aggregates": { "avg_temp": { "avg": 78.1 } } }, { "value": "HIGH", "count": 238, "aggregates": { "avg_temp": { "avg": 91.4 } } }, { "value": "CRITICAL", "count": 77, "aggregates": { "avg_temp": { "avg": 118.6 } } } ] } } } ``` #### Example: grouping through a record view The same distribution, expressed with [view property paths](/cdf/dm/records/concepts/records_and_streams#property-paths-through-views). `EquipmentEvent/v1` maps the `equipment_events` container; because the aggregation references the view, only records with data in all containers the view maps are counted, and any filter defined on the view applies too. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "aggregates": { "by_severity": { "uniqueValues": { "property": ["factory-data", "EquipmentEvent/v1", "severity"], "aggregates": { "avg_temp": { "avg": { "property": ["factory-data", "EquipmentEvent/v1", "temperature"] } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples view_path = "EquipmentEvent/v1" response = client.post( url=f"/api/v1/projects/{client.config.project}/streams/{stream_id}/records/aggregate", json={ "lastUpdatedTime": {"gt": "2025-10-01T00:00:00.000Z"}, "aggregates": { "by_severity": { "uniqueValues": { "property": [space_id, view_path, "severity"], "aggregates": { "avg_temp": { "avg": {"property": [space_id, view_path, "temperature"]} } }, } } }, }, ) result = response.json() for bucket in result["aggregates"]["by_severity"]["uniqueValueBuckets"]: print(f"{bucket['value']}: {bucket['count']} events, avg temp {bucket['aggregates']['avg_temp']['avg']}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const viewPath = 'EquipmentEvent/v1'; const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, aggregates: { by_severity: { uniqueValues: { property: [spaceId, viewPath, 'severity'], aggregates: { avg_temp: { avg: { property: [spaceId, viewPath, 'temperature'] }, }, }, }, }, }, }); result.by_severity.uniqueValueBuckets.forEach((bucket) => { console.log(`${bucket.value}: ${bucket.count} events, avg temp ${bucket.aggregates.avg_temp.avg}`); }); ``` ### numberHistogram Divides numeric values into fixed-width intervals (buckets) and counts how many records fall into each interval. Each bucket represents a range starting at the bucket key and extending to the next interval boundary. For example, with an interval of 10, a bucket with key `60.0` contains records with values in the range \[60, 70). The bucket key for a value is calculated as: ``` bucket_key = floor(value / interval) * interval ``` This means each value is rounded down to its closest interval boundary. A record with a temperature of `73.2` and an interval of `20` would be assigned to the bucket with key `60.0` (since floor(73.2 / 20) \* 20 = 60), which covers the range \[60, 80). `numberHistogram` is similar to binning a numeric column and grouping by the bins in SQL or pandas/Polars: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT FLOOR(temperature / 20) * 20 AS bin, COUNT(*) FROM equipment_events GROUP BY bin ORDER BY bin ``` In pandas, this is `df.groupby(df["temperature"] // 20 * 20).size()`, and in Polars, `df.group_by(pl.col("temperature").floordiv(20) * 20).len()`. #### Parameters | Parameter | Type | Required | Description | | - | - | - | - | | `property` | array | Yes | The numeric property to bucket. Supports `int32`, `int64`, `float32`, `float64`. | | `interval` | number | Yes | The width of each bucket. Must be a positive number. | | `hardBounds` | object | No | Limits the range of buckets. Has optional `min` (inclusive) and `max` (inclusive) fields. Only buckets whose keys fall within \[min, max] are returned, even if matching records exist outside these bounds. | | `aggregates` | object | No | Nested sub-aggregates to compute within each bucket. | **Empty buckets** By default, the response includes empty buckets (buckets with a count of 0) between the lowest and highest values in the data. This provides a continuous, gap-free view of the data distribution. For example, if your data has values at 20 and 80 with an interval of 20, you will get buckets for 20, 40, 60, and 80, even if the 40 and 60 buckets have a count of 0. The first bucket is determined by the smallest value in the data, and the last bucket by the largest value. **Hard bounds vs. filters** Use `hardBounds` to limit which buckets appear in the response. This is purely a display-level control, it does not affect which records are aggregated, only which buckets are included in the output. Records outside the bounds are still counted if they fall into a bucket that starts within the bounds. This is different from using a `filter` in the request, which restricts which records are considered for the entire aggregation. For example, if you set `hardBounds: { "min": 40, "max": 120 }`, records with temperatures below 40 are still counted (in the bucket whose range starts below 40 if it overlaps with the bounds), but the response only includes buckets whose keys fall within \[40, 120] (both bounds inclusive). If you only need results for a specific range, prefer a `filter` over `hardBounds`. Filters reduce the number of records the service needs to aggregate, which improves performance. Reserve `hardBounds` for cases where you need multiple views in a single request, for example, a full-range metric aggregate alongside bucketed results for a narrower range. **Missing values** Records that do not have the specified property (or have a `null` value) are excluded from the histogram. They do not contribute to any bucket. #### Example: temperature distribution ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "temp_distribution": { "numberHistogram": { "property": ["factory-data", "equipment_events", "temperature"], "interval": 20, "hardBounds": { "min": 0, "max": 160 } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import NumberHistogram result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "temp_distribution": NumberHistogram( property=[space_id, container_id, "temperature"], interval=20, hard_bounds={"min": 0, "max": 160}, ), }, ) for bucket in result["temp_distribution"].buckets: print(f"[{bucket.interval_start}, {bucket.interval_start + 20}): {bucket.count} events") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { temp_distribution: { numberHistogram: { property: [spaceId, containerId, 'temperature'], interval: 20, hardBounds: { min: 0, max: 160 }, }, }, }, }); const buckets = result.temp_distribution.numberHistogramBuckets; buckets.forEach((bucket) => { console.log(`[${bucket.intervalStart}, ${bucket.intervalStart + 20}): ${bucket.count} events`); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "temp_distribution": { "numberHistogramBuckets": [ { "intervalStart": 0, "count": 12 }, { "intervalStart": 20, "count": 45 }, { "intervalStart": 40, "count": 178 }, { "intervalStart": 60, "count": 412 }, { "intervalStart": 80, "count": 356 }, { "intervalStart": 100, "count": 168 }, { "intervalStart": 120, "count": 0 }, { "intervalStart": 140, "count": 62 }, { "intervalStart": 160, "count": 17 } ] } } } ``` ### timeHistogram Divides timestamp values into time-based intervals and counts records per interval. This is the primary aggregation for building time-series charts, trend lines, and temporal analysis. Each bucket represents a time window starting at the `intervalStart` timestamp. `timeHistogram` is similar to truncating timestamps and grouping by the truncated value in SQL or pandas/Polars: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT DATE_TRUNC('hour', recorded_at) AS hour, COUNT(*) FROM equipment_events GROUP BY hour ORDER BY hour ``` In pandas, this is `df.resample("1h", on="recorded_at").size()`, and in Polars, `df.group_by_dynamic("recorded_at", every="1h").agg(pl.len())`. You must specify exactly one of `calendarInterval` or `fixedInterval`: * **calendarInterval**: Uses calendar-aware intervals that respect natural time boundaries. For example, `1M` produces monthly buckets where each month has its actual number of days (28, 29, 30, or 31), and `1d` respects daylight saving time transitions. Only single-unit values are allowed because calendar units vary in length (a "2-month" interval would be ambiguous since months have different numbers of days). * **fixedInterval**: Uses fixed-duration intervals measured in consistent time units. Any multiplier is allowed (e.g., `30m`, `6h`, `2d`). Each bucket has exactly the same duration in absolute time. Use fixed intervals when you need precise, uniform bucket sizes. For example, `6h` always means exactly 21,600,000 milliseconds, regardless of daylight saving time or calendar boundaries. Choose `calendarInterval` when you want human-readable time boundaries (start of each day, month, or year). Choose `fixedInterval` when you need precise, uniform durations (every 30 minutes, every 6 hours). #### Parameters | Parameter | Type | Required | Description | | - | - | - | - | | `property` | array | Yes | The timestamp property to bucket. Also supports `["createdTime"]` and `["lastUpdatedTime"]`. | | `calendarInterval` | string | One of | Calendar-aware interval. Allowed values: `1s` (second), `1m` (minute), `1h` (hour), `1d` (day), `1w` (week), `1M` (month), `1q` (quarter), `1y` (year). | | `fixedInterval` | string | One of | Fixed-duration interval as a duration string. Supported units: `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours), `d` (days). Any positive multiplier is allowed (e.g., `30m`, `6h`, `2d`). | | `hardBounds` | object | No | Limits the range of time buckets. Has optional `min` (inclusive) and `max` (exclusive) fields. Values must be ISO 8601 date-time strings with timezone (e.g., `2025-06-15T00:00:00Z`, `2025-06-15T00:00:00.000Z`, or `2025-06-15T02:00:00+02:00`). Epoch milliseconds are not supported. Only buckets whose keys fall within \[min, max) are returned. | | `aggregates` | object | No | Nested sub-aggregates to compute within each bucket. | You must specify **exactly one** of `calendarInterval` or `fixedInterval`. Providing both or neither causes a validation error. For `calendarInterval`, only single-unit multipliers are allowed (`1h`, `1d`, `1M`, etc.). Multiples like `2d` or `6h` are **not valid** for calendar intervals because calendar units have variable lengths. Use `fixedInterval` instead if you need non-unit intervals (e.g., `fixedInterval: "6h"`). The following details describe how the `timeHistogram` handles edge cases such as empty intervals, bounded time ranges, and missing data. **Empty buckets** Like `numberHistogram`, the `timeHistogram` returns empty buckets (count of 0) between the earliest and latest data points. This ensures a continuous time series with no gaps, which is important for charting and trend analysis. For example, if your data has records at 08:00 and 14:00 with a `calendarInterval` of `"1h"`, you will get buckets for every hour from 08:00 through 14:00, including hours with no records. **Hard bounds for time ranges** Use `hardBounds` with `min` (inclusive) and `max` (exclusive) as ISO 8601 date-time strings to limit which time buckets appear in the response. Only buckets whose keys fall within \[min, max) are returned. Supported formats include `2025-06-15T00:00:00Z`, `2025-06-15T00:00:00.000Z`, and `2025-06-15T02:00:00+02:00` (timezone offsets are converted to UTC). Epoch milliseconds are not supported; use ISO 8601 date-time strings instead. Note that `timeHistogram` uses a half-open range \[min, max) while `numberHistogram` uses a closed range \[min, max]. Like `numberHistogram`, `hardBounds` controls only which buckets are returned — it does not filter which records are aggregated. **Missing values** Records that do not have the specified timestamp property (or have a `null` value) are excluded from the time histogram. They do not contribute to any time bucket. #### Example: hourly event counts ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "hourly_events": { "timeHistogram": { "property": ["factory-data", "equipment_events", "recorded_at"], "calendarInterval": "1h" } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import TimeHistogram result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "hourly_events": TimeHistogram( property=[space_id, container_id, "recorded_at"], calendar_interval="1h", ), }, ) for bucket in result["hourly_events"].buckets: print(f"{bucket.interval_start}: {bucket.count} events") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { hourly_events: { timeHistogram: { property: [spaceId, containerId, 'recorded_at'], calendarInterval: '1h', }, }, }, }); const buckets = result.hourly_events.timeHistogramBuckets; buckets.forEach((bucket) => { console.log(`${bucket.intervalStart}: ${bucket.count} events`); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "hourly_events": { "timeHistogramBuckets": [ { "intervalStart": "2025-10-24T08:00:00.000Z", "count": 45 }, { "intervalStart": "2025-10-24T09:00:00.000Z", "count": 78 }, { "intervalStart": "2025-10-24T10:00:00.000Z", "count": 112 }, { "intervalStart": "2025-10-24T11:00:00.000Z", "count": 0 }, { "intervalStart": "2025-10-24T12:00:00.000Z", "count": 63 }, { "intervalStart": "2025-10-24T13:00:00.000Z", "count": 88 }, { "intervalStart": "2025-10-24T14:00:00.000Z", "count": 102 }, { "intervalStart": "2025-10-24T15:00:00.000Z", "count": 71 }, { "intervalStart": "2025-10-24T16:00:00.000Z", "count": 34 } ] } } } ``` ### filters Creates one bucket per filter expression, where each bucket contains the count of records matching that filter. Use this aggregation to segment records into categories defined by arbitrary conditions. For example, splitting events into severity bands based on numeric ranges, or comparing events across different equipment types. The `filters` aggregate is similar to conditional counting in a single query in SQL or pandas/Polars: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT COUNT(*) FILTER (WHERE temperature < 50) AS cold, COUNT(*) FILTER (WHERE temperature >= 50 AND temperature < 80) AS normal, COUNT(*) FILTER (WHERE temperature >= 80 AND temperature < 110) AS warm, COUNT(*) FILTER (WHERE temperature >= 110) AS hot FROM equipment_events ``` In pandas or Polars, this is equivalent to applying multiple boolean masks and counting each subset. Note that unlike `GROUP BY`, a record can match multiple filters and appear in more than one bucket. Each filter uses the same [filter syntax](/cdf/dm/dm_concepts/dm_querying) available on the filter and sync endpoints. #### Parameters | Parameter | Type | Required | Description | | - | - | - | - | | `filters` | array | Yes | An array of 1-10 filter expressions. Each filter defines one bucket. | | `aggregates` | object | No | Nested sub-aggregates to compute within each bucket. | **Bucket ordering** Buckets are returned in the same order as the filters in your request. Since the filters are provided as an array (not a named map), you identify each bucket by its position. For example, the first bucket in the `filterBuckets` response array corresponds to the first filter in your `filters` array. **Overlapping filters** A record can match multiple filters and appear in more than one bucket. The filters are evaluated independently, they do not partition the data into mutually exclusive groups. If you need non-overlapping segments, design your filter conditions so they don't overlap (for example, use `lt` and `gte` to create adjacent ranges without gaps or overlaps). **Limits** A single request can have at most **30 filter buckets** across all `filters` aggregates combined. For example, if you have three `filters` aggregates with 10 filters each, that uses all 30. #### Example: temperature severity bands ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "temp_bands": { "filters": { "filters": [ { "range": { "property": ["factory-data", "equipment_events", "temperature"], "lt": 50 } }, { "range": { "property": ["factory-data", "equipment_events", "temperature"], "gte": 50, "lt": 80 } }, { "range": { "property": ["factory-data", "equipment_events", "temperature"], "gte": 80, "lt": 110 } }, { "range": { "property": ["factory-data", "equipment_events", "temperature"], "gte": 110 } } ] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import Filters temperature = [space_id, container_id, "temperature"] result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "temp_bands": Filters( filters=[ {"range": {"property": temperature, "lt": 50}}, {"range": {"property": temperature, "gte": 50, "lt": 80}}, {"range": {"property": temperature, "gte": 80, "lt": 110}}, {"range": {"property": temperature, "gte": 110}}, ] ), }, ) labels = ["Cold (<50)", "Normal (50-80)", "Warm (80-110)", "Hot (>110)"] for i, bucket in enumerate(result["temp_bands"].buckets): print(f"{labels[i]}: {bucket.count} events") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { temp_bands: { filters: { filters: [ { range: { property: [spaceId, containerId, 'temperature'], lt: 50 } }, { range: { property: [spaceId, containerId, 'temperature'], gte: 50, lt: 80 } }, { range: { property: [spaceId, containerId, 'temperature'], gte: 80, lt: 110 } }, { range: { property: [spaceId, containerId, 'temperature'], gte: 110 } }, ], }, }, }, }); const labels = ['Cold (<50)', 'Normal (50-80)', 'Warm (80-110)', 'Hot (>110)']; result.temp_bands.filterBuckets.forEach((bucket, i) => { console.log(`${labels[i]}: ${bucket.count} events`); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "temp_bands": { "filterBuckets": [ { "count": 57 }, { "count": 590 }, { "count": 524 }, { "count": 79 } ] } } } ``` The four buckets correspond to the four filters in order: below 50 degrees Celsius, 50-80 degrees Celsius, 80-110 degrees Celsius, and above 110 degrees Celsius. ## Pipeline aggregates Pipeline aggregates compute derived values from the output of bucket aggregations. Unlike metric and bucket aggregates that operate on records directly, pipeline aggregates process the results of other aggregates, making them ideal for trend smoothing, rate calculations, and comparative analysis. ### movingFunction Slides a window across the buckets of a histogram and applies a function to the values within that window. This is commonly used to smooth noisy data by computing a moving average over time, or to calculate rolling sums and extremes. `movingFunction` is similar to SQL window functions with a frame clause, `.rolling()` in pandas, or `.rolling_mean()` in Polars: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT hour, event_count, AVG(event_count) OVER (ORDER BY hour ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS moving_avg FROM hourly_counts ``` In pandas, this is `df["event_count"].rolling(window=3).mean()`, and in Polars, `df.select(pl.col("event_count").rolling_mean(window_size=3))`. The `movingFunction` aggregate **must be nested inside** a `numberHistogram` or `timeHistogram`. It cannot be used as a standalone aggregate. #### Parameters | Parameter | Type | Required | Description | | - | - | - | - | | `bucketsPath` | string | Yes | Path to the metric to use as input. Use `_count` for the bucket record count, or the name of a sibling metric aggregate. | | `window` | integer | Yes | Size of the sliding window in number of buckets. Must be at least 1. | | `function` | string | Yes | The function to apply over the window. | #### Available functions | Function | Python SDK | Description | | - | - | - | | `MovingFunctions.unweightedAvg` | `MovingFunctions.UNWEIGHTED_AVG` | Simple average of all values in the window. The most common choice for trend smoothing. | | `MovingFunctions.linearWeightedAvg` | `MovingFunctions.LINEAR_WEIGHTED_AVG` | Weighted average where more recent values have higher weight. Gives a trend line that responds faster to recent changes. | | `MovingFunctions.sum` | `MovingFunctions.SUM` | Sum of all values in the window. Useful for calculating rolling totals. | | `MovingFunctions.min` | `MovingFunctions.MIN` | Minimum value in the window. | | `MovingFunctions.max` | `MovingFunctions.MAX` | Maximum value in the window. | In the Cognite Python SDK, pass a `MovingFunctions` enum member rather than a string. An invalid name then raises an error locally instead of reaching the API. The enum also accepts the API string, with or without the `MovingFunctions.` prefix. Both `MovingFunctions("MovingFunctions.sum")` and `MovingFunctions("sum")` resolve to `MovingFunctions.SUM`. **Window warm-up** The first few buckets in the response will have a window smaller than the specified `window` size because there aren't enough preceding buckets yet. For example, with `window: 3`: * Bucket 1: the `movingFunction` receives only one value (the current bucket), so `MovingFunctions.unweightedAvg` returns 0 because it requires at least 2 values. * Bucket 2: the window contains 2 values (buckets 1 and 2), so the average is computed from those two. * Bucket 3 onward: the full 3-bucket window is used. This warm-up behavior means the first few results may not be representative. For `MovingFunctions.unweightedAvg`, the first bucket always returns `0` as the `fnValue`. **Handling gaps in data** When a bucket in the window has no data (a gap), the function skips that bucket and computes the result from the remaining values in the window. This prevents empty time periods from distorting the smoothed trend line. For example, if a 3-bucket window covers hours 10:00, 11:00, and 12:00, but 11:00 has no records, the moving average is computed from only the 10:00 and 12:00 values. **Pipeline vs. sub-aggregate** Pipeline aggregates like `movingFunction` are fundamentally different from sub-aggregates. Sub-aggregates (like nesting `avg` inside `timeHistogram`) operate on the records within each bucket. Pipeline aggregates operate on the computed results of other aggregates across buckets — they process bucket-level outputs, not individual records. #### bucketsPath syntax The `bucketsPath` specifies which value from the parent histogram buckets to feed into the function. Paths are relative to the parent histogram aggregate, not absolute paths from the root of the request. The formal syntax is: ``` AGG_SEPARATOR = `>` ; AGG_NAME = ; MULTIBUCKET_KEY = `[]` KEY_NAME = ; PATH = ?()* ; ``` Common path patterns: | Path | Description | | - | - | | `_count` | The record count of each bucket. Use this when you want to apply the moving function to the number of records per bucket. | | `my_metric` | The result of a sibling metric aggregate named `my_metric`. For example, if you have a sibling `avg` aggregate named `avg_temp`, use `"bucketsPath": "avg_temp"`. | | `my_agg>nested_metric` | A metric inside a nested aggregate. Use `>` to traverse levels. For example, `"by_severity>avg_temp"` accesses the `avg_temp` metric inside the `by_severity` bucket aggregate. | | `my_agg[bucket_key]>metric` | A metric inside a specific bucket of a multi-bucket aggregate. For example, `"by_severity[HIGH]>avg_temp"` accesses the average temperature for the "HIGH" severity bucket only. | #### Example: 3-hour moving average of event counts ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "hourly_trend": { "timeHistogram": { "property": ["factory-data", "equipment_events", "recorded_at"], "calendarInterval": "1h", "aggregates": { "moving_avg_3h": { "movingFunction": { "bucketsPath": "_count", "window": 3, "function": "MovingFunctions.unweightedAvg" } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples from cognite.client.data_classes.data_modeling.aggregates import MovingFunction, MovingFunctions result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "hourly_trend": TimeHistogram( property=[space_id, container_id, "recorded_at"], calendar_interval="1h", aggregates={ "moving_avg_3h": MovingFunction( buckets_path="_count", window=3, function=MovingFunctions.UNWEIGHTED_AVG, ), }, ), }, ) for bucket in result["hourly_trend"].buckets: print(f"{bucket.interval_start}: {bucket.count} events, moving avg: {bucket.aggregates['moving_avg_3h'].fn_value}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { hourly_trend: { timeHistogram: { property: [spaceId, containerId, 'recorded_at'], calendarInterval: '1h', aggregates: { moving_avg_3h: { movingFunction: { bucketsPath: '_count', window: 3, function: 'MovingFunctions.unweightedAvg', }, }, }, }, }, }, }); const buckets = result.hourly_trend.timeHistogramBuckets; buckets.forEach((bucket) => { console.log(`${bucket.intervalStart}: ${bucket.count} events, moving avg: ${bucket.aggregates.moving_avg_3h.fnValue}`); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "hourly_trend": { "timeHistogramBuckets": [ { "intervalStart": "2025-10-24T08:00:00.000Z", "count": 45, "aggregates": { "moving_avg_3h": { "fnValue": 45.0 } } }, { "intervalStart": "2025-10-24T09:00:00.000Z", "count": 78, "aggregates": { "moving_avg_3h": { "fnValue": 61.5 } } }, { "intervalStart": "2025-10-24T10:00:00.000Z", "count": 112, "aggregates": { "moving_avg_3h": { "fnValue": 78.3 } } }, { "intervalStart": "2025-10-24T11:00:00.000Z", "count": 95, "aggregates": { "moving_avg_3h": { "fnValue": 95.0 } } }, { "intervalStart": "2025-10-24T12:00:00.000Z", "count": 63, "aggregates": { "moving_avg_3h": { "fnValue": 90.0 } } }, { "intervalStart": "2025-10-24T13:00:00.000Z", "count": 88, "aggregates": { "moving_avg_3h": { "fnValue": 82.0 } } }, { "intervalStart": "2025-10-24T14:00:00.000Z", "count": 102, "aggregates": { "moving_avg_3h": { "fnValue": 84.3 } } }, { "intervalStart": "2025-10-24T15:00:00.000Z", "count": 71, "aggregates": { "moving_avg_3h": { "fnValue": 87.0 } } }, { "intervalStart": "2025-10-24T16:00:00.000Z", "count": 34, "aggregates": { "moving_avg_3h": { "fnValue": 69.0 } } } ] } } } ``` The bar chart shows raw hourly counts while the line shows the smoothed 3-hour moving average, making it easier to identify the underlying trend. ## Nesting aggregates Bucket aggregates (`uniqueValues`, `numberHistogram`, `timeHistogram`, and `filters`) can contain nested sub-aggregates of any type — including other bucket aggregates. This lets you build multi-dimensional analyses in a single API call. ### How nesting works When you add sub-aggregates to a bucket aggregate, the sub-aggregates are computed independently within each bucket. For example, nesting an `avg` inside a `uniqueValues` aggregate computes a separate average for each unique value group. You can nest aggregates up to 5 levels deep, with up to 5 aggregates per level and 16 total aggregates in the request. Nesting maps directly to multi-column `GROUP BY` with aggregate functions in SQL or pandas/Polars. The example below groups by `location`, then by `severity` within each location, computing an average temperature per group: ```sql SQL theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT location, severity, COUNT(*) AS count, AVG(temperature) AS avg_temp FROM equipment_events GROUP BY location, severity ORDER BY location, count DESC ``` ```python Polars theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} df.group_by("location", "severity").agg( pl.len().alias("count"), pl.col("temperature").mean().alias("avg_temp"), ).sort("location", "count", descending=[False, True]) ``` ```python pandas theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} df.groupby(["location", "severity"]).agg( count=("temperature", "size"), avg_temp=("temperature", "mean"), ).sort_values(["location", "count"], ascending=[True, False]) ``` In the Records API, this becomes a `uniqueValues` on `location` with a nested `uniqueValues` on `severity` and a nested `avg` on `temperature`, all in a single request. While nesting is powerful, deep nesting with high-cardinality data can be expensive. Keep these performance considerations in mind. Deep nesting with high-cardinality bucket aggregates can be expensive in terms of computation and memory. For example, nesting a `uniqueValues` with `size: 1000` inside another `uniqueValues` with `size: 1000` could produce up to 1,000,000 bucket combinations. A single aggregate request can produce a maximum of **10,000 buckets** across all aggregations. If the total number of buckets exceeds this limit, the API returns a **400 Bad Request** error. Note that even requests that stay under the limit can consume significant resources if the bucket count is high. To keep requests efficient: * Use smaller `size` values for outer bucket aggregates when possible. * Place the highest-cardinality groupings at the deepest level of nesting, not the outermost. * Prefer metric aggregates (like `count`, `avg`, `sum`) as leaf-level sub-aggregates rather than adding more bucket levels. * If you need more than 16 aggregates or deeper nesting, split the analysis across multiple requests. ### Example: equipment breakdown by location and severity This example uses two levels of `uniqueValues` with a nested `avg` to answer: "For each location, what are the top severity levels and their average temperatures?" ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "by_location": { "uniqueValues": { "property": ["factory-data", "equipment_events", "location"], "size": 5, "aggregates": { "by_severity": { "uniqueValues": { "property": ["factory-data", "equipment_events", "severity"] } }, "avg_temp": { "avg": { "property": ["factory-data", "equipment_events", "temperature"] } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "by_location": UniqueValues( property=[space_id, container_id, "location"], size=5, aggregates={ "by_severity": UniqueValues(property=[space_id, container_id, "severity"]), "avg_temp": Average(property=[space_id, container_id, "temperature"]), }, ), }, ) for loc in result["by_location"].buckets: print(f"{loc.value}: {loc.count} events, avg temp {loc.aggregates['avg_temp'].value}") for sev in loc.aggregates["by_severity"].buckets: print(f" {sev.value}: {sev.count}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { by_location: { uniqueValues: { property: [spaceId, containerId, 'location'], size: 5, aggregates: { by_severity: { uniqueValues: { property: [spaceId, containerId, 'severity'], }, }, avg_temp: { avg: { property: [spaceId, containerId, 'temperature'] }, }, }, }, }, }, }); const locations = result.by_location.uniqueValueBuckets; locations.forEach((loc) => { console.log(`${loc.value}: ${loc.count} events, avg temp ${loc.aggregates.avg_temp.avg}`); loc.aggregates.by_severity.uniqueValueBuckets.forEach((sev) => { console.log(` ${sev.value}: ${sev.count}`); }); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "by_location": { "uniqueValueBuckets": [ { "value": "Building-A", "count": 680, "aggregates": { "avg_temp": { "avg": 74.2 }, "by_severity": { "uniqueValueBuckets": [ { "value": "LOW", "count": 290 }, { "value": "MEDIUM", "count": 215 }, { "value": "HIGH", "count": 130 }, { "value": "CRITICAL", "count": 45 } ] } } }, { "value": "Building-B", "count": 570, "aggregates": { "avg_temp": { "avg": 70.1 }, "by_severity": { "uniqueValueBuckets": [ { "value": "LOW", "count": 233 }, { "value": "MEDIUM", "count": 197 }, { "value": "HIGH", "count": 108 }, { "value": "CRITICAL", "count": 32 } ] } } } ] } } } ``` ### Example: time-series with severity bands and trend line This example combines `timeHistogram`, `filters`, and `movingFunction` to create a stacked severity breakdown over time with a smoothed trend line — all in a single request. This is a common pattern for alarm monitoring dashboards. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-01T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "factory-data", "externalId": "equipment_events" }] }, "aggregates": { "events_over_time": { "timeHistogram": { "property": ["factory-data", "equipment_events", "recorded_at"], "calendarInterval": "1d", "aggregates": { "by_severity_band": { "filters": { "filters": [ { "equals": { "property": ["factory-data", "equipment_events", "severity"], "value": "LOW" } }, { "equals": { "property": ["factory-data", "equipment_events", "severity"], "value": "MEDIUM" } }, { "or": [ { "equals": { "property": ["factory-data", "equipment_events", "severity"], "value": "HIGH" } }, { "equals": { "property": ["factory-data", "equipment_events", "severity"], "value": "CRITICAL" } } ] } ] } }, "trend": { "movingFunction": { "bucketsPath": "_count", "window": 3, "function": "MovingFunctions.unweightedAvg" } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Using client and variables from previous examples severity = [space_id, container_id, "severity"] result = client.data_modeling.records.aggregate( stream_id=stream_id, last_updated_time=TimeRange(gt="2025-10-01T00:00:00.000Z"), filter={"hasData": [{"type": "container", "space": space_id, "externalId": container_id}]}, aggregates={ "events_over_time": TimeHistogram( property=[space_id, container_id, "recorded_at"], calendar_interval="1d", aggregates={ "by_severity_band": Filters( filters=[ {"equals": {"property": severity, "value": "LOW"}}, {"equals": {"property": severity, "value": "MEDIUM"}}, {"or": [ {"equals": {"property": severity, "value": "HIGH"}}, {"equals": {"property": severity, "value": "CRITICAL"}}, ]}, ] ), "trend": MovingFunction( buckets_path="_count", window=3, function=MovingFunctions.UNWEIGHTED_AVG, ), }, ), }, ) band_labels = ["LOW", "MEDIUM", "HIGH+CRITICAL"] for bucket in result["events_over_time"].buckets: print(f"{bucket.interval_start}: {bucket.count} events, trend: {bucket.aggregates['trend'].fn_value}") for i, band in enumerate(bucket.aggregates["by_severity_band"].buckets): print(f" {band_labels[i]}: {band.count}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Using client and variables from previous examples const result = await client.records.aggregate(streamId, { lastUpdatedTime: { gt: '2025-10-01T00:00:00.000Z' }, filter: { hasData: [{ type: 'container' as const, space: spaceId, externalId: containerId }], }, aggregates: { events_over_time: { timeHistogram: { property: [spaceId, containerId, 'recorded_at'], calendarInterval: '1d', aggregates: { by_severity_band: { filters: { filters: [ { equals: { property: [spaceId, containerId, 'severity'], value: 'LOW', }, }, { equals: { property: [spaceId, containerId, 'severity'], value: 'MEDIUM', }, }, { or: [ { equals: { property: [spaceId, containerId, 'severity'], value: 'HIGH', }, }, { equals: { property: [spaceId, containerId, 'severity'], value: 'CRITICAL', }, }, ], }, ], }, }, trend: { movingFunction: { bucketsPath: '_count', window: 3, function: 'MovingFunctions.unweightedAvg', }, }, }, }, }, }, }); const buckets = result.events_over_time.timeHistogramBuckets; const bandLabels = ['LOW', 'MEDIUM', 'HIGH+CRITICAL']; buckets.forEach((bucket) => { console.log(`${bucket.intervalStart}: ${bucket.count} events, trend: ${bucket.aggregates.trend.fnValue}`); bucket.aggregates.by_severity_band.filterBuckets.forEach((band, i) => { console.log(` ${bandLabels[i]}: ${band.count}`); }); }); ``` ```json Example response theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "events_over_time": { "timeHistogramBuckets": [ { "intervalStart": "2025-10-21T00:00:00.000Z", "count": 180, "aggregates": { "by_severity_band": { "filterBuckets": [ { "count": 85 }, { "count": 62 }, { "count": 33 } ] }, "trend": { "fnValue": 180.0 } } }, { "intervalStart": "2025-10-22T00:00:00.000Z", "count": 210, "aggregates": { "by_severity_band": { "filterBuckets": [ { "count": 95 }, { "count": 72 }, { "count": 43 } ] }, "trend": { "fnValue": 195.0 } } }, { "intervalStart": "2025-10-23T00:00:00.000Z", "count": 165, "aggregates": { "by_severity_band": { "filterBuckets": [ { "count": 78 }, { "count": 58 }, { "count": 29 } ] }, "trend": { "fnValue": 185.0 } } }, { "intervalStart": "2025-10-24T00:00:00.000Z", "count": 195, "aggregates": { "by_severity_band": { "filterBuckets": [ { "count": 88 }, { "count": 68 }, { "count": 39 } ] }, "trend": { "fnValue": 190.0 } } } ] } } } ``` In this response, each day's bucket contains three severity bands (LOW, MEDIUM, HIGH+CRITICAL) and a 3-day moving average trend. The `filterBuckets` are ordered to match the filter array: index 0 is LOW, index 1 is MEDIUM, and index 2 is HIGH+CRITICAL. ## Limits and constraints The aggregate endpoint enforces limits on request complexity to ensure consistent performance. These limits apply to the aggregate tree structure, individual aggregate parameters, and the overall request. ### Aggregate tree limits | Constraint | Value | | - | - | | Max aggregates per level | 5 | | Max aggregate tree depth | 5 | | Max total aggregates in request | 16 | | Max total buckets per request | 10,000 | | Aggregate name length | 1-255 characters | | Aggregate name pattern | Cannot contain `[`, `]`, or `>` | ### Aggregate-specific limits | Aggregate | Constraint | Value | | - | - | - | | `uniqueValues` | `size` range | 1-2000 (default: 10) | | `filters` | Filters per aggregate | 1-10 | | `filters` | Total filter buckets across all `filters` aggregates | 30 | | `movingFunction` | Minimum `window` | 1 | | `movingFunction` | `bucketsPath` length | 1-1280 characters | | `timeHistogram` | `calendarInterval` values | `1s` (second), `1m` (minute), `1h` (hour), `1d` (day), `1w` (week), `1M` (month), `1q` (quarter), `1y` (year) | | `timeHistogram` | `fixedInterval` units | `ms`, `s`, `m`, `h`, `d` (any positive multiplier, e.g., `30m`, `6h`) | | `timeHistogram` | `fixedInterval` length | 1-100 characters | ### General request limits | Constraint | Value | | - | - | | `lastUpdatedTime` | Required for immutable streams, optional for mutable | | `maxFilteringInterval` | Maximum time range between `gt` and `lt` in `lastUpdatedTime`. Defined by the stream template (e.g., `P1Y` for one year). Retrieve from stream settings. | | Rate limits | Aggregate requests have stricter rate limits than filter or sync requests. See [API rate limits](/api-reference/concepts/20230101/rate-limits). | If you need more than 16 aggregates or deeper nesting, split your analysis across multiple requests. Each request gets a consistent snapshot of the data, so results from separate requests made close together will be comparable. ## Common patterns ### Trend analysis Combine `timeHistogram` with `movingFunction` to smooth noisy time-series data and identify trends: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "daily_trend": { "timeHistogram": { "property": ["factory-data", "equipment_events", "recorded_at"], "calendarInterval": "1d", "aggregates": { "avg_temp": { "avg": { "property": ["factory-data", "equipment_events", "temperature"] } }, "smoothed_temp": { "movingFunction": { "bucketsPath": "avg_temp", "window": 7, "function": "MovingFunctions.unweightedAvg" } } } } } } } ``` This computes a 7-day moving average of daily temperatures, smoothing out day-to-day variation to reveal the underlying trend. ### Distribution analysis Use `uniqueValues` with nested metrics to understand how a categorical dimension relates to numeric measures: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "equipment_analysis": { "uniqueValues": { "property": ["factory-data", "equipment_events", "equipment_id"], "size": 50, "aggregates": { "event_count": { "count": {} }, "avg_priority": { "avg": { "property": ["factory-data", "equipment_events", "priority"] } }, "max_temp": { "max": { "property": ["factory-data", "equipment_events", "temperature"] } } } } } } } ``` ### Threshold monitoring Use `filters` to count records in specific value ranges for threshold-based alerting: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "aggregates": { "status_check": { "filters": { "filters": [ { "and": [ { "equals": { "property": ["factory-data", "equipment_events", "is_critical"], "value": true } }, { "equals": { "property": ["factory-data", "equipment_events", "severity"], "value": "CRITICAL" } } ] }, { "range": { "property": ["factory-data", "equipment_events", "temperature"], "gte": 120 } } ] } } } } ``` ## Further reading * [About records and streams](/cdf/dm/records/concepts/records_and_streams) — Core concepts for records, streams, and containers * [Get started with Records](/cdf/dm/records/guides/records_getting_started) — Hands-on tutorial for setting up schemas, streams, and queries * [Filter syntax](/cdf/dm/dm_concepts/dm_querying) — Filter expressions used with the `filter` parameter and the `filters` aggregate * [Limits and restrictions](/cdf/dm/dm_reference/dm_limits_and_restrictions#records-and-streams) — Stream counts, template limits, and resource limits for records # Get started with Records Source: https://docs.cognite.com/cdf/dm/records/guides/records_getting_started Learn Records core concepts and patterns through a hands-on tutorial. Follow along with a SCADA alarm management example that demonstrates schemas, streams, ingestion, querying, and stream-to-stream pipelines. This hands-on guide teaches you the core patterns for working with Records in Cognite Data Fusion (CDF). You'll learn how to create schemas, set up streams, ingest high-volume event data, query records efficiently, and build data pipelines between streams. Throughout this guide, **SCADA alarm management** serves as a practical example, but these patterns apply to any high-volume industrial data: multi-value sensor readings, event frames, equipment logs, telemetry events, and more. The Records service is built specifically for ingesting billions of events and querying them efficiently. This tutorial teaches Records patterns and APIs; it does not cover capacity planning. Before you create production streams or ingest at scale, read and understand [Choose and size streams](/cdf/dm/records/guides/choose_and_size_streams). Use that guide to confirm records fit your workload, choose stream templates, and plan storage and retention. Records are designed primarily for **immutable data** like logs, events, and historical records that don't change after ingestion. If your data needs updates during its lifecycle (like active alarms awaiting acknowledgment), you can use a mutable stream as a transitional stage, and then archive the finalized records to an immutable stream. For data that requires frequent updates and will never be archived, consider using [data modeling nodes](/cdf/dm/dm_concepts/dm_spaces_instances#instance) instead. By the end of this guide, you'll have a working example that handles the full lifecycle: ingesting data, updating records during their active phase, querying efficiently, and archiving completed records to immutable storage. If you're new to Records concepts like spaces, containers, and streams, see [Records and streams](/cdf/dm/records/concepts/records_and_streams) for background. ## Prerequisites 1. Configure [access control](/cdf/access/guides/capabilities#records) for your user or service account. 2. Development environment with either: * Python with `cognite-sdk` installed, or * Node.js/TypeScript with `@cognite/sdk` installed ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes import GroupWrite from cognite.client.data_classes.capabilities import ( DataModelsAcl, StreamRecordsAcl, StreamsAcl, ) client = CogniteClient() my_group = client.iam.groups.create( GroupWrite( name="Records Access Group", capabilities=[ DataModelsAcl( actions=[ DataModelsAcl.Action.Read, DataModelsAcl.Action.Write ], scope=DataModelsAcl.Scope.All(), ), StreamRecordsAcl( actions=[ StreamRecordsAcl.Action.Read, StreamRecordsAcl.Action.Write ], scope=StreamRecordsAcl.Scope.All(), ), StreamsAcl( actions=[ StreamsAcl.Action.Read, StreamsAcl.Action.Create, StreamsAcl.Action.Delete, ], scope=StreamsAcl.Scope.All(), ), ], ) ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const myGroup = await client.groups.create([{ name: 'Records Access Group', capabilities: [ { dataModelsAcl: { actions: ['READ', 'WRITE'], scope: { all: {} } } }, { streamRecordsAcl: { actions: ['READ', 'WRITE'], scope: { all: {} } } }, { streamsAcl: { actions: ['READ', 'CREATE', 'DELETE'], scope: { all: {} } } } ] }]); ``` ## Build your first Records pipeline In this section, you'll build a complete data pipeline using a SCADA alarm management scenario as an example. This pipeline demonstrates the full Records lifecycle: using a mutable stream as a temporary staging area for active data, ingesting and updating records, querying with different endpoints, and archiving finalized records to an immutable stream for permanent storage. This pattern mirrors what you'd deploy in production for managing any high-volume event data where records have a defined lifecycle. Start by defining the schema for your alarm records. You'll create a [space](/cdf/dm/dm_concepts/dm_spaces_instances#space) to organize your data and a [container](/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) that defines the structure of each alarm event. If you're new to these concepts, check out the [data modeling concepts](/cdf/dm/dm_concepts/dm_containers_views_datamodels) for background. 1. **Create a space** to serve as a namespace for your SCADA alarm data. This keeps your alarm schemas and records organized and separate from other data in your CDF project. ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling import SpaceApply client = CogniteClient() space_id = "scada-alarms" space = client.data_modeling.spaces.apply( SpaceApply( space=space_id, description="SCADA alarm events", name="SCADA Alarms", ) ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const space = await client.spaces.upsert([ { space: spaceId, description: 'SCADA alarm events', name: 'SCADA Alarms', } ]); ``` 2. **Create and define the container** that specifies what properties each alarm record will have. This schema includes the fields you'd typically see in SCADA systems: alarm identifiers, severity levels, numeric values, priorities, acknowledgment status, and timestamps. When creating containers for records, you must set `usedFor` to `"record"`. Records can only be ingested into containers specifically designated for records. ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling import ( ContainerApply, ContainerProperty, Text, Float64, Int64, Boolean, Timestamp, ) client = CogniteClient() container_id = "alarm_events" container = client.data_modeling.containers.apply( ContainerApply( space="scada-alarms", external_id=container_id, name="SCADA Alarm Events", description="Container for SCADA alarm event data", used_for="record", properties={ # Text property - alarm identifier "alarm_id": ContainerProperty( type=Text(is_list=False), nullable=False, name="Alarm ID", description="Unique identifier for the alarm", ), # Text property - alarm severity "severity": ContainerProperty( type=Text(is_list=False), nullable=False, name="Severity", description="Alarm severity level (CRITICAL, HIGH, MEDIUM, LOW)", ), # Float property - alarm value "value": ContainerProperty( type=Float64(is_list=False), nullable=False, name="Value", description="Numeric value associated with the alarm condition", ), # Int property - priority score "priority": ContainerProperty( type=Int64(is_list=False), nullable=False, name="Priority", description="Priority score (1-100)", ), # Boolean property - acknowledged status "is_acknowledged": ContainerProperty( type=Boolean(is_list=False), nullable=False, name="Acknowledged Status", description="True if alarm has been acknowledged", ), # Timestamp property - when alarm occurred "timestamp": ContainerProperty( type=Timestamp(is_list=False), nullable=False, name="Timestamp", description="Time when the alarm occurred", ), }, ) ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const containerId = 'alarm_events'; const container = await client.containers.upsert([ { space: 'scada-alarms', externalId: containerId, name: 'SCADA Alarm Events', description: 'Container for SCADA alarm event data', usedFor: 'record', properties: { alarm_id: { type: { type: 'text' }, nullable: false, name: 'Alarm ID', description: 'Unique identifier for the alarm', }, severity: { type: { type: 'text' }, nullable: false, name: 'Severity', description: 'Alarm severity level (CRITICAL, HIGH, MEDIUM, LOW)', }, value: { type: { type: 'float64' }, nullable: false, name: 'Value', description: 'Numeric value associated with the alarm condition', }, priority: { type: { type: 'int64' }, nullable: false, name: 'Priority', description: 'Priority score (1-100)', }, is_acknowledged: { type: { type: 'boolean' }, nullable: false, name: 'Acknowledged Status', description: 'True if alarm has been acknowledged', }, timestamp: { type: { type: 'timestamp' }, nullable: false, name: 'Timestamp', description: 'Time when the alarm occurred', }, }, } ]); ``` Now create the streams that will hold your alarm records. Streams need a [stream settings template](/cdf/dm/records/concepts/records_and_streams#stream-templates) that defines their behavior, performance characteristics, and lifecycle policies. For this pipeline, you'll create two streams to handle different stages of the alarm lifecycle. Choose your stream template based on **data lifecycle and throughput requirements**, not by record type or source system. For the limit values of each template, see [Stream template limits](/cdf/dm/dm_reference/dm_limits_and_restrictions#stream-template-limits). * **Create a mutable stream** for active alarms that need real-time updates, like when an operator acknowledges an alarm. This stream uses the `BasicLiveData` template, which is optimized for data that changes frequently. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "items": [ { "externalId": "alarm_data_live_stream", "settings": {"template": {"name": "BasicLiveData"}} } ] }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.streams import ( StreamTemplate, StreamTemplateWriteSettings, StreamWrite, ) client = CogniteClient() mutable_stream_id = "alarm_data_live_stream" stream = client.data_modeling.streams.create( StreamWrite( external_id=mutable_stream_id, settings=StreamTemplateWriteSettings(template=StreamTemplate(name="BasicLiveData")), ) ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const mutableStreamId = 'alarm_data_live_stream'; const response = await client.streams.create({ externalId: mutableStreamId, settings: { template: { name: 'BasicLiveData', }, }, }); ``` * **Create an immutable stream** for archived alarms that won't change after they're written. This stream uses the `BasicArchive` template and is designed for write-once, read-many scenarios and long-term retention. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "items": [ { "externalId": "alarm_data_archive_stream", "settings": {"template": {"name": "BasicArchive"}} } ] }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.streams import ( StreamTemplate, StreamTemplateWriteSettings, StreamWrite, ) client = CogniteClient() immutable_stream_id = "alarm_data_archive_stream" stream = client.data_modeling.streams.create( StreamWrite( external_id=immutable_stream_id, settings=StreamTemplateWriteSettings(template=StreamTemplate(name="BasicArchive")), ) ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const immutableStreamId = 'alarm_data_archive_stream'; const response = await client.streams.create({ externalId: immutableStreamId, settings: { template: { name: 'BasicArchive', }, }, }); ``` Now you'll ingest alarm records into your mutable stream. In production, you'd typically use [a Cognite extractor](/cdf/integration/concepts/extraction#cognite-extractors) that reads directly from your source system and writes to Records. For this tutorial, you'll use the API directly to simulate the ingestion process. The Records service does **not** support onboarding data using RAW and Transformations. Always ingest data directly to a Records stream using one of the supported extractors: * [OPC UA Extractor](/cdf/integration/guides/extraction/opc_ua) (Immutable stream support) * [PI AF Extractor](/cdf/integration/guides/extraction/pi_af) * [Hosted REST Extractor](/cdf/integration/guides/extraction/rest) When writing records to the API, you need to specify the target stream identifier and include the container's `space`.`externalId` to identify which schema the data follows. 1. **Start by ingesting a couple of hardcoded alarm records** to verify your setup works. This gives you a small dataset to test queries and updates before adding more data. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "items": [ { "space": "scada-alarms", "externalId": "record-001", "sources": [{ "source": { "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }, "properties": { "alarm_id": "ALARM-PRESS-001", "severity": "CRITICAL", "value": 125.8, "priority": 95, "is_acknowledged": false, "timestamp": "2025-10-22T10:00:00.000Z" } }] }, { "space": "scada-alarms", "externalId": "record-002", "sources": [{ "source": { "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }, "properties": { "alarm_id": "ALARM-TEMP-042", "severity": "MEDIUM", "value": 88.3, "priority": 60, "is_acknowledged": false, "timestamp": "2025-10-22T10:00:00.000Z" } }] } ] }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordSource, RecordWrite, ) import uuid client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" # Create two hardcoded alarm records hardcoded_records = [ { "alarm_id": "ALARM-PRESS-001", "severity": "CRITICAL", "value": 125.8, "priority": 95, "is_acknowledged": False, "timestamp": "2025-10-22T10:00:00.000Z", }, { "alarm_id": "ALARM-TEMP-042", "severity": "MEDIUM", "value": 88.3, "priority": 60, "is_acknowledged": False, "timestamp": "2025-10-22T10:00:00.000Z", }, ] # Prepare records for ingestion container = RecordContainerId(space=space_id, external_id=container_id) items = [ RecordWrite( space=space_id, external_id=str(uuid.uuid4()), sources=[RecordSource(source=container, properties=record)], ) for record in hardcoded_records ] # Ingest hardcoded records client.data_modeling.records.ingest(items, stream_id=mutable_stream_id) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Create two hardcoded alarm records const hardcodedRecords = [ { alarm_id: 'ALARM-PRESS-001', severity: 'CRITICAL', value: 125.8, priority: 95, is_acknowledged: false, timestamp: '2025-10-22T10:00:00.000Z', }, { alarm_id: 'ALARM-TEMP-042', severity: 'MEDIUM', value: 88.3, priority: 60, is_acknowledged: false, timestamp: '2025-10-22T10:00:00.000Z', }, ]; // Prepare records for ingestion const items = hardcodedRecords.map((record) => ({ space: spaceId, externalId: crypto.randomUUID(), sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: record, }, ], })); // Ingest hardcoded records await client.records.ingest(mutableStreamId, items); ``` 2. **Add 100 random alarm records** to simulate a realistic dataset. This larger dataset will make your queries and aggregations more meaningful and help you see how Records handles volume. ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordSource, RecordWrite, ) from datetime import datetime, timedelta import random import uuid client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" # Generate 100 random alarm records alarm_ids = [ "ALARM-PRESS-001", "ALARM-TEMP-042", "ALARM-FLOW-013", "ALARM-VIBR-088", "ALARM-LEVEL-055", ] severities = ["CRITICAL", "HIGH", "MEDIUM", "LOW"] base_time = datetime.now() - timedelta(hours=2) random_records = [ { "alarm_id": random.choice(alarm_ids), "severity": random.choice(severities), "value": round(random.uniform(50.0, 150.0), 2), "priority": random.randint(20, 100), "is_acknowledged": random.choice([True, False]), "timestamp": (base_time + timedelta(minutes=i)).strftime("%Y-%m-%dT%H:%M:%S.000Z"), } for i in range(100) ] # Prepare random records for ingestion container = RecordContainerId(space=space_id, external_id=container_id) items = [ RecordWrite( space=space_id, external_id=str(uuid.uuid4()), sources=[RecordSource(source=container, properties=record)], ) for record in random_records ] # Ingest random records client.data_modeling.records.ingest(items, stream_id=mutable_stream_id) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Generate 100 random alarm records const alarmIds = [ 'ALARM-PRESS-001', 'ALARM-TEMP-042', 'ALARM-FLOW-013', 'ALARM-VIBR-088', 'ALARM-LEVEL-055', ]; const severities = ['CRITICAL', 'HIGH', 'MEDIUM', 'LOW']; const baseTime = new Date(Date.now() - 2 * 60 * 60 * 1000); // 2 hours ago const randomRecords = Array.from({ length: 100 }, (_, i) => { const timestamp = new Date(baseTime.getTime() + i * 60 * 1000); return { alarm_id: alarmIds[Math.floor(Math.random() * alarmIds.length)], severity: severities[Math.floor(Math.random() * severities.length)], value: Math.round((50 + Math.random() * 100) * 100) / 100, priority: Math.floor(20 + Math.random() * 81), is_acknowledged: Math.random() > 0.5, timestamp: timestamp.toISOString(), }; }); // Prepare random records for ingestion const items = randomRecords.map((record) => ({ space: spaceId, externalId: crypto.randomUUID(), sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: record, }, ], })); // Ingest random records await client.records.ingest(mutableStreamId, items); ``` One advantage of mutable streams is the ability to update records after they're written. This is essential for scenarios like acknowledging alarms when an operator responds, updating status fields as conditions change, or correcting data errors. Use the `upsert` endpoint to update existing records by their `space` and `externalId`. Records do **not** support partial updates. When you update a record, you must provide **all** properties in the container, not just the ones you want to change. All non-nullable properties must be included in every upsert request. Omitting one causes a validation error. See [Updating records in mutable streams](/cdf/dm/records/concepts/records_and_streams#updating-records-in-mutable-streams) for more information. The recommended workflow for updating a record: 1. **Read** the existing record using the `filter` or `sync` endpoint. 2. **Merge** your changes into the full property set. 3. **Upsert** the complete record back to the stream. To update a record, you need its `externalId`. In practice, you'd typically get this from a previous query or store it when you first ingest the record. For this example, you'll update one of the alarm records you ingested earlier to mark it as acknowledged. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/upsert" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "items": [ { "space": "scada-alarms", "externalId": "your-record-external-id", "sources": [{ "source": { "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }, "properties": { "alarm_id": "ALARM-TEMP-042", "severity": "MEDIUM", "value": 88.3, "priority": 60, "is_acknowledged": true, "timestamp": "2025-10-22T10:00:00.000Z" } }] } ] }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordSource, RecordWrite, ) client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" # Assuming you have the external_id of a record you want to update # In practice, you would get this from a previous query or store it during ingestion record_external_id = "your-record-external-id" # Replace with actual ID # Update the record to set is_acknowledged to True updated_record = { "alarm_id": "ALARM-TEMP-042", "severity": "MEDIUM", "value": 88.3, "priority": 60, "is_acknowledged": True, "timestamp": "2025-10-22T10:00:00.000Z", } client.data_modeling.records.upsert( RecordWrite( space=space_id, external_id=record_external_id, sources=[ RecordSource( source=RecordContainerId(space=space_id, external_id=container_id), properties=updated_record, ) ], ), stream_id=mutable_stream_id, ) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Assuming you have the external_id of a record you want to update // In practice, you would get this from a previous query or store it during ingestion const recordExternalId = 'your-record-external-id'; // Replace with actual ID // Update the record to set is_acknowledged to true const updatedRecord = { alarm_id: 'ALARM-TEMP-042', severity: 'MEDIUM', value: 88.3, priority: 60, is_acknowledged: true, timestamp: '2025-10-22T10:00:00.000Z', }; await client.records.upsert(mutableStreamId, [ { space: spaceId, externalId: recordExternalId, sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: updatedRecord, }, ], }, ]); ``` Next, you'll query your alarm records using the Records API. The API provides three endpoints for consuming data, each optimized for different use cases: * For **interactive queries**, use the `filter` endpoint. * For **large datasets with pagination**, use the `sync` endpoint. * For **statistical analysis**, use the `aggregate` endpoint. The query syntax for filtering is similar across all endpoints. When querying records, we recommend that you use the [`hasData` filter](/cdf/dm/dm_concepts/dm_querying#hasdata-filter) to ensure you only retrieve records that contain data in the container you're querying. A record can exist in a stream but may not have data for every container. The `hasData` filter ensures efficient queries by excluding records that don't have data in the selected container. **Time-based filtering with lastUpdatedTime** The `lastUpdatedTime` filter is **mandatory** for `filter` and `aggregate` queries on immutable streams. This filter defines the time range for retrieving records: * **`gt` (greater than)**: The start of the time range (required for immutable streams) * **`lt` (less than)**: The end of the time range (optional, defaults to current time if not specified) The maximum time range you can query **in a single request** is defined by the stream's `maxFilteringInterval` setting, which is determined by the [stream template](/cdf/dm/records/concepts/records_and_streams#stream-templates) you chose when the stream was created. This limit applies to the span between `gt` and `lt`, not to how far back in time you can reach, as you can query any historical period as long as each request stays within the interval. To query data spanning more than this interval, split your requests into adjacent time windows. See [Query time range limits](/cdf/dm/records/concepts/records_and_streams#query-time-range-limits) for details and examples. For mutable streams, `lastUpdatedTime` is optional, but using it improves query performance. * **Use the `filter` endpoint** for interactive applications, dashboards, and any scenario where you expect a relatively small, bounded result set. It returns an unpaginated result set with a maximum of 1,000 records per request and supports custom sorting on any property. This endpoint is ideal for queries like "show me the top 10 unacknowledged alarms sorted by priority." ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/filter" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-23T00:00:00.000Z" }, "sources": [{ "source": { "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }, "properties": ["*"] }], "filter": { "and": [ { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, { "equals": { "property": ["scada-alarms", "alarm_events", "is_acknowledged"], "value": false } } ] }, "sort": [{ "property": ["scada-alarms", "alarm_events", "priority"], "direction": "descending" }], "limit": 10 }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling import InstanceSort from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordSourceSelector, TimeRange, ) from cognite.client.data_classes.filters import And, Equals, HasData from datetime import datetime, timedelta from tabulate import tabulate client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" container = RecordContainerId(space=space_id, external_id=container_id) # Filter for unacknowledged alarms, sorted by priority descending records = client.data_modeling.records.filter( stream_id=mutable_stream_id, last_updated_time=TimeRange( gt=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%S.00Z"), ), sources=[RecordSourceSelector(source=container, properties=["*"])], filter=And( HasData(containers=[container]), Equals(property=[space_id, container_id, "is_acknowledged"], value=False), ), sort=InstanceSort(property=[space_id, container_id, "priority"], direction="descending"), limit=10, ) # Display as table table_data = [] for record in records[:5]: # Show first 5 props = record.properties[space_id][container_id] table_data.append([ props["alarm_id"], props["severity"], props["value"], props["priority"], props["timestamp"], ]) print(f"Found {len(records)} unacknowledged alarms (showing first 5):") print(tabulate( table_data, headers=["Alarm ID", "Severity", "Value", "Priority", "Timestamp"], tablefmt="github", )) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Filter for unacknowledged alarms, sorted by priority descending const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); const filterResults = await client.records.filter(mutableStreamId, { lastUpdatedTime: { gt: oneDayAgo, }, sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: ['*'], }, ], filter: { and: [ { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, { equals: { property: [spaceId, containerId, 'is_acknowledged'], value: false, }, }, ], }, sort: [ { property: [spaceId, containerId, 'priority'], direction: 'descending', }, ], limit: 10, }); // Display first 5 results console.log(`Found ${filterResults.length} unacknowledged alarms (showing first 5):`); filterResults.slice(0, 5).forEach((item) => { const props = item.properties[spaceId][containerId]; console.log(`${props.alarm_id} | ${props.severity} | ${props.value} | ${props.priority} | ${props.timestamp}`); }); ``` **Expected output:** ``` Found 10 unacknowledged alarms (showing first 5): | Alarm ID | Severity | Value | Priority | Timestamp | |-----------------|------------|---------|------------|--------------------------| | ALARM-TEMP-042 | CRITICAL | 98.28 | 100 | 2025-10-24T12:04:54.000Z | | ALARM-TEMP-042 | LOW | 71.82 | 100 | 2025-10-24T12:44:54.000Z | | ALARM-LEVEL-055 | LOW | 69.19 | 99 | 2025-10-24T12:55:54.000Z | | ALARM-FLOW-013 | LOW | 56.17 | 97 | 2025-10-24T12:50:54.000Z | | ALARM-FLOW-013 | HIGH | 131.5 | 97 | 2025-10-24T11:45:54.000Z | ``` * **Use the `sync` endpoint** for batch processing, data exports, or any application that needs to retrieve a large number of records efficiently. It returns a paginated result set—you iterate through the data by making subsequent requests with the cursor returned in the previous response. The sync process is complete when the API response has `hasNext` set to `false`. The sync endpoint supports a maximum of 1,000 records per page and doesn't support custom sorting. The read throughput is governed by the stream settings defined by the [stream template](/cdf/dm/records/concepts/records_and_streams#stream-templates) you chose when creating the stream. This endpoint is designed for workflows like "export all recent alarm events for analysis" or for implementing incremental data pipelines. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/sync" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "sources": [{ "source": { "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }, "properties": ["*"] }], "filter": { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, "initializeCursor": "2m-ago", "limit": 10 }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordSourceSelector, ) from cognite.client.data_classes.filters import HasData from tabulate import tabulate client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" container = RecordContainerId(space=space_id, external_id=container_id) # Sync records from the last 2 minutes page = client.data_modeling.records.sync( stream_id=mutable_stream_id, sources=[RecordSourceSelector(source=container, properties=["*"])], filter=HasData(containers=[container]), initialize_cursor="2m-ago", limit=10, ) # Display as table table_data = [] for record in page[:5]: # Show first 5 props = record.properties[space_id][container_id] ack_status = "ACK" if props["is_acknowledged"] else "UNACK" table_data.append([ props["alarm_id"], props["severity"], ack_status, props["value"], props["timestamp"], ]) print(f"Sync returned {len(page)} records (showing first 5):") print(tabulate( table_data, headers=["Alarm ID", "Severity", "Status", "Value", "Timestamp"], tablefmt="github", )) if page.cursor: cursor_preview = f"{page.cursor[:5]}..." if len(page.cursor) > 5 else page.cursor print(f"\nNext cursor for incremental sync: {cursor_preview}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Sync records from the last 2 minutes const syncResponse = await client.records .sync(mutableStreamId, { sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: ['*'], }, ], filter: { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, initializeCursor: '2m-ago', limit: 10, }); const syncItems = syncResponse.items; // Display first 5 results console.log(`Sync returned ${syncItems.length} records (showing first 5):`); syncItems.slice(0, 5).forEach((item) => { const props = item.properties[spaceId][containerId]; const ackStatus = props.is_acknowledged ? 'ACK' : 'UNACK'; console.log(`${props.alarm_id} | ${props.severity} | ${ackStatus} | ${props.value} | ${props.timestamp}`); }); if (syncResponse.nextCursor) { const cursorPreview = syncResponse.nextCursor.length > 5 ? `${syncResponse.nextCursor.substring(0, 5)}...` : syncResponse.nextCursor; console.log(`\nNext cursor for incremental sync: ${cursorPreview}`); } ``` **Expected output:** ``` Sync returned 10 records (showing first 5): | Alarm ID | Severity | Status | Value | Timestamp | |-----------------|------------|----------|---------|--------------------------| | ALARM-PRESS-001 | CRITICAL | UNACK | 125.8 | 2025-10-22T10:00:00.000Z | | ALARM-TEMP-042 | LOW | UNACK | 98.78 | 2025-10-24T11:27:54.000Z | | ALARM-VIBR-088 | CRITICAL | ACK | 122.78 | 2025-10-24T11:28:54.000Z | | ALARM-VIBR-088 | MEDIUM | ACK | 140.56 | 2025-10-24T11:29:54.000Z | | ALARM-LEVEL-055 | CRITICAL | ACK | 50.59 | 2025-10-24T11:30:54.000Z | Next cursor for incremental sync: 08c0e... ``` * **Use the `aggregate` endpoint** to perform statistical analysis on your records, such as counts, averages, sums, and other aggregations. It returns aggregated results based on the specified aggregation functions and groupings, optimized for analytical queries that require summarizing large datasets without retrieving individual records. This endpoint is ideal for generating reports like "count of alarms by severity" or "average alarm value and priority statistics." ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-23T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, "aggregates": { "total_count": { "count": {} }, "avg_value": { "avg": { "property": ["scada-alarms", "alarm_events", "value"] } }, "min_value": { "min": { "property": ["scada-alarms", "alarm_events", "value"] } }, "max_value": { "max": { "property": ["scada-alarms", "alarm_events", "value"] } }, "avg_priority": { "avg": { "property": ["scada-alarms", "alarm_events", "priority"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.aggregates import Average, Count, Max, Min from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, TimeRange, ) from cognite.client.data_classes.filters import HasData from datetime import datetime, timedelta client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" container = RecordContainerId(space=space_id, external_id=container_id) # Calculate overall statistics: count, average value, min/max values, average priority res = client.data_modeling.records.aggregate( stream_id=mutable_stream_id, last_updated_time=TimeRange( gt=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%S.00Z"), ), filter=HasData(containers=[container]), aggregates={ "total_count": Count(), "avg_value": Average([space_id, container_id, "value"]), "min_value": Min([space_id, container_id, "value"]), "max_value": Max([space_id, container_id, "value"]), "avg_priority": Average([space_id, container_id, "priority"]), }, ) print("Overall Statistics:") print(f" Total Records: {res['total_count'].value}") print(f" Average Value: {res['avg_value'].value:.2f}") print(f" Min Value: {res['min_value'].value:.2f}") print(f" Max Value: {res['max_value'].value:.2f}") print(f" Average Priority: {res['avg_priority'].value:.1f}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Calculate overall statistics const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); const aggResults = await client.records.aggregate(mutableStreamId, { lastUpdatedTime: { gt: oneDayAgo, }, filter: { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, aggregates: { total_count: { count: {} }, avg_value: { avg: { property: [spaceId, containerId, 'value'] } }, min_value: { min: { property: [spaceId, containerId, 'value'] } }, max_value: { max: { property: [spaceId, containerId, 'value'] } }, avg_priority: { avg: { property: [spaceId, containerId, 'priority'] } }, }, }); console.log('Overall Statistics:'); console.log(` Total Records: ${aggResults.total_count.count}`); console.log(` Average Value: ${aggResults.avg_value.avg.toFixed(2)}`); console.log(` Min Value: ${aggResults.min_value.min.toFixed(2)}`); console.log(` Max Value: ${aggResults.max_value.max.toFixed(2)}`); console.log(` Average Priority: ${aggResults.avg_priority.avg.toFixed(1)}`); ``` **Expected output:** ``` Overall Statistics: Total Records: 102 Average Value: 95.87 Min Value: 50.15 Max Value: 149.39 Average Priority: 63.0 ``` ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/${STREAM_ID}/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-23T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, "aggregates": { "by_alarm": { "uniqueValues": { "property": ["scada-alarms", "alarm_events", "alarm_id"], "aggregates": { "avg_value": { "avg": { "property": ["scada-alarms", "alarm_events", "value"] } }, "by_severity": { "uniqueValues": { "property": ["scada-alarms", "alarm_events", "severity"] } } } } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.aggregates import Average, UniqueValues from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, TimeRange, ) from cognite.client.data_classes.filters import HasData from datetime import datetime, timedelta from tabulate import tabulate client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" container = RecordContainerId(space=space_id, external_id=container_id) # Group by alarm_id and calculate statistics for each alarm res = client.data_modeling.records.aggregate( stream_id=mutable_stream_id, last_updated_time=TimeRange( gt=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%S.00Z"), ), filter=HasData(containers=[container]), aggregates={ "by_alarm": UniqueValues( property=[space_id, container_id, "alarm_id"], aggregates={ "avg_value": Average([space_id, container_id, "value"]), "by_severity": UniqueValues(property=[space_id, container_id, "severity"]), }, ) }, ) print("Per-Alarm Statistics:") table_data = [] for bucket in res["by_alarm"].buckets: alarm = bucket.value count = bucket.count avg_value = bucket.aggregates["avg_value"].value # Get most common severity from nested uniqueValues buckets severity_buckets = bucket.aggregates["by_severity"].buckets most_common_severity = ( max(severity_buckets, key=lambda b: b.count).value if severity_buckets else "N/A" ) table_data.append([alarm, count, f"{avg_value:.2f}", most_common_severity]) print(tabulate( table_data, headers=["Alarm ID", "Occurrences", "Avg Value", "Most Common Severity"], tablefmt="github", )) ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; // Group by alarm_id and calculate statistics for each alarm const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); const groupbyResults = await client.records.aggregate(mutableStreamId, { lastUpdatedTime: { gt: oneDayAgo, }, filter: { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, aggregates: { by_alarm: { uniqueValues: { property: [spaceId, containerId, 'alarm_id'], aggregates: { avg_value: { avg: { property: [spaceId, containerId, 'value'] }, }, by_severity: { uniqueValues: { property: [spaceId, containerId, 'severity'], }, }, }, }, }, }, }); console.log('Per-Alarm Statistics:'); const buckets = groupbyResults.by_alarm.uniqueValueBuckets; buckets.forEach((bucket) => { const alarm = bucket.value; const count = bucket.count; const avgValue = bucket.aggregates.avg_value.avg; // Get most common severity from nested uniqueValues buckets const severityBuckets = bucket.aggregates.by_severity.uniqueValueBuckets; const mostCommonSeverity = severityBuckets.length > 0 ? severityBuckets.reduce((max, b) => (b.count > max.count ? b : max)).value : 'N/A'; console.log(`${alarm} | ${count} | ${avgValue.toFixed(2)} | ${mostCommonSeverity}`); }); ``` **Expected output:** ``` Per-Alarm Statistics: | Alarm ID | Occurrences | Avg Value | Most Common Severity | |-----------------|---------------|-------------|------------------------| | ALARM-FLOW-013 | 26 | 91.50 | LOW | | ALARM-TEMP-042 | 24 | 92.15 | CRITICAL | | ALARM-PRESS-001 | 19 | 104.62 | CRITICAL | | ALARM-VIBR-088 | 17 | 106.93 | LOW | | ALARM-LEVEL-055 | 16 | 86.39 | CRITICAL | ``` The final step of this tutorial is to implement a common Records pattern: moving data between streams based on lifecycle stages. You'll archive acknowledged alarms by moving them from the mutable stream to the immutable stream. This stream-to-stream pattern is useful for: * **Hot/cold storage**: keep active data in mutable streams, archive historical data in immutable streams * **Data lifecycle management**: move processed records to long-term storage automatically * **Performance optimization**: keep active streams lean by moving old data to archives The key to reliable stream-to-stream operations is using the `sync` endpoint with cursor-based pagination. The cursor tracks your position in the stream, allowing you to resume from where you left off if processing fails or is interrupted. ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, RecordId, RecordSource, RecordSourceSelector, RecordWrite, ) from cognite.client.data_classes.filters import Equals client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" immutable_stream_id = "alarm_data_archive_stream" container = RecordContainerId(space=space_id, external_id=container_id) # In a production system, you would persist this cursor between runs stored_cursor = None # Pagination settings BATCH_SIZE = 10 total_records_moved = 0 batch_number = 0 print("Starting pagination loop to archive acknowledged alarms...") # Keep paginating until there are no more acknowledged records while True: batch_number += 1 print(f"\n--- Batch {batch_number} ---") # Sync acknowledged records from MUTABLE stream. # Use sync_resume with the stored cursor if available, otherwise initialize from 2 minutes ago. # A record without data in this container has no is_acknowledged value, so this equality # check alone already excludes it; hasData would be redundant here. if stored_cursor: page = client.data_modeling.records.sync_resume( stream_id=mutable_stream_id, cursor=stored_cursor, sources=[RecordSourceSelector(source=container, properties=["*"])], filter=Equals(property=[space_id, container_id, "is_acknowledged"], value=True), limit=BATCH_SIZE, ) else: page = client.data_modeling.records.sync( stream_id=mutable_stream_id, initialize_cursor="2m-ago", sources=[RecordSourceSelector(source=container, properties=["*"])], filter=Equals(property=[space_id, container_id, "is_acknowledged"], value=True), limit=BATCH_SIZE, ) acknowledged_records = list(page) next_cursor = page.cursor has_next = page.has_next print(f"Found {len(acknowledged_records)} acknowledged records in this batch") print(f"More data immediately available: {has_next}") # If no records found, we're done if len(acknowledged_records) == 0: print("No more acknowledged records to move") stored_cursor = next_cursor # Update cursor even when no records found break # Prepare records for ingestion into immutable stream print(f"Ingesting {len(acknowledged_records)} records to IMMUTABLE stream...") immutable_items = [] records_to_delete = [] for record in acknowledged_records: # Extract properties and identifiers props = record.properties[space_id][container_id] # Store record identifier for deletion records_to_delete.append(RecordId(space=record.space, external_id=record.external_id)) # Prepare for immutable stream immutable_items.append( RecordWrite( space=record.space, external_id=record.external_id, sources=[RecordSource(source=container, properties=props)], ) ) # Ingest into immutable stream (raises on failure) client.data_modeling.records.ingest(immutable_items, stream_id=immutable_stream_id) print(f"✓ Ingested {len(immutable_items)} records to IMMUTABLE stream") # Delete acknowledged records from MUTABLE stream print(f"Deleting {len(records_to_delete)} records from MUTABLE stream...") client.data_modeling.records.delete(records_to_delete, stream_id=mutable_stream_id) print(f"✓ Deleted {len(records_to_delete)} records from MUTABLE stream") # IMPORTANT: Only update cursor after BOTH operations succeed stored_cursor = next_cursor total_records_moved += len(acknowledged_records) cursor_preview = f"{stored_cursor[:20]}..." if len(stored_cursor) > 20 else stored_cursor print(f"✓ Cursor updated: {cursor_preview}") print(f"Total records moved so far: {total_records_moved}") print(f"\n{'=' * 70}") print("PAGINATION COMPLETE") print(f"{'=' * 70}") print(f"Total batches processed: {batch_number}") print(f"Total records moved: {total_records_moved}") print(f"{'=' * 70}") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; const immutableStreamId = 'alarm_data_archive_stream'; // In a production system, you would persist this cursor between runs let storedCursor: string | null = null; // Pagination settings const BATCH_SIZE = 10; let totalRecordsMoved = 0; let batchNumber = 0; console.log(`Starting pagination loop (batch size: ${BATCH_SIZE} records)...`); // Keep paginating until there are no more acknowledged records while (true) { batchNumber++; console.log(`\n--- Batch ${batchNumber} ---`); const syncResult = await client.records .sync(mutableStreamId, { sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: ['*'], }, ], // Use stored cursor if available, otherwise initialize from 2 minutes ago ...(storedCursor ? { cursor: storedCursor } : { initializeCursor: '2m-ago' }), // A record without data in this container has no is_acknowledged value, so this // equality check alone already excludes it; hasData would be redundant here. filter: { equals: { property: [spaceId, containerId, 'is_acknowledged'], value: true, }, }, limit: BATCH_SIZE, }); const acknowledgedRecords = syncResult.items; const nextCursor = syncResult.nextCursor; const hasNext = syncResult.hasNext; console.log(`Found ${acknowledgedRecords.length} acknowledged records in this batch`); console.log(`More data immediately available: ${hasNext}`); // If no records found, we're done if (acknowledgedRecords.length === 0) { console.log('No more acknowledged records to move'); storedCursor = nextCursor; break; } // Prepare acknowledged records for ingestion into immutable stream const immutableItems = []; const recordsToDelete = []; for (const record of acknowledgedRecords) { const recordSpace = record.space; const recordExternalId = record.externalId; const props = record.properties![spaceId][containerId]; recordsToDelete.push({ space: recordSpace, externalId: recordExternalId, }); immutableItems.push({ space: recordSpace, externalId: recordExternalId, sources: [ { source: { type: 'container' as const, space: spaceId, externalId: containerId, }, properties: props, }, ], }); } // Ingest into immutable stream console.log(`Ingesting ${immutableItems.length} records to IMMUTABLE stream...`); await client.records.ingest(immutableStreamId, immutableItems); console.log(`✓ Ingested ${immutableItems.length} records to IMMUTABLE stream`); // Delete from mutable stream console.log(`Deleting ${recordsToDelete.length} records from MUTABLE stream...`); await client.records.delete(mutableStreamId, recordsToDelete); console.log(`✓ Deleted ${recordsToDelete.length} records from MUTABLE stream`); // IMPORTANT: Only update the cursor after BOTH operations succeed // This ensures that if either ingest or delete fails, we can retry from the same position storedCursor = nextCursor; totalRecordsMoved += acknowledgedRecords.length; const cursorPreview = storedCursor.length > 20 ? `${storedCursor.substring(0, 20)}...` : storedCursor; console.log(`✓ Cursor updated: ${cursorPreview}`); console.log(`Total records moved so far: ${totalRecordsMoved}`); } console.log('\n' + '='.repeat(70)); console.log('PAGINATION COMPLETE'); console.log('='.repeat(70)); console.log(`Total batches processed: ${batchNumber}`); console.log(`Total records moved: ${totalRecordsMoved}`); console.log(`Batch size used: ${BATCH_SIZE}`); console.log('='.repeat(70)); ``` **Expected output:** ``` Starting pagination loop to archive acknowledged alarms... --- Batch 1 --- Found 10 acknowledged records in this batch More data immediately available: True Ingesting 10 records to IMMUTABLE stream... ✓ Ingested 10 records to IMMUTABLE stream Deleting 10 records from MUTABLE stream... ✓ Deleted 10 records from MUTABLE stream ✓ Cursor updated: 08c0b89cecbeb1dab818... Total records moved so far: 10 --- Batch 2 --- Found 10 acknowledged records in this batch More data immediately available: True Ingesting 10 records to IMMUTABLE stream... ✓ Ingested 10 records to IMMUTABLE stream Deleting 10 records from MUTABLE stream... ✓ Deleted 10 records from MUTABLE stream ✓ Cursor updated: 08c0aae0c3c0b1dab818... Total records moved so far: 20 --- Batch 3 --- Found 10 acknowledged records in this batch More data immediately available: True Ingesting 10 records to IMMUTABLE stream... ✓ Ingested 10 records to IMMUTABLE stream Deleting 10 records from MUTABLE stream... ✓ Deleted 10 records from MUTABLE stream ✓ Cursor updated: 08c0e5b4cdc4b1dab818... Total records moved so far: 30 [... batches 4-6 similar, each moving 10 records ...] --- Batch 7 --- Found 4 acknowledged records in this batch More data immediately available: False Ingesting 4 records to IMMUTABLE stream... ✓ Ingested 4 records to IMMUTABLE stream Deleting 4 records from MUTABLE stream... ✓ Deleted 4 records from MUTABLE stream ✓ Cursor updated: 0880beeae1c9b1dab818... Total records moved so far: 64 --- Batch 8 --- Found 0 acknowledged records in this batch More data immediately available: False No more acknowledged records to move ====================================================================== PAGINATION COMPLETE ====================================================================== Total batches processed: 8 Total records moved: 64 ====================================================================== ``` After archiving, verify that the records were moved correctly by querying both streams to see the final distribution of acknowledged and unacknowledged alarms. ```bash curl theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Query mutable stream curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/alarm_data_live_stream/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-23T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, "aggregates": { "total_count": { "count": {} }, "by_acknowledged": { "uniqueValues": { "property": ["scada-alarms", "alarm_events", "is_acknowledged"] } } } }' # Query immutable stream curl -X POST \ "https://${CLUSTER}.cognitedata.com/api/v1/projects/${PROJECT}/streams/alarm_data_archive_stream/records/aggregate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TOKEN}" \ -d '{ "lastUpdatedTime": { "gt": "2025-10-23T00:00:00.000Z" }, "filter": { "hasData": [{ "type": "container", "space": "scada-alarms", "externalId": "alarm_events" }] }, "aggregates": { "total_count": { "count": {} }, "by_acknowledged": { "uniqueValues": { "property": ["scada-alarms", "alarm_events", "is_acknowledged"] } } } }' ``` ```python Python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.aggregates import Count, UniqueValues from cognite.client.data_classes.data_modeling.records import ( RecordContainerId, TimeRange, ) from cognite.client.data_classes.filters import HasData from datetime import datetime, timedelta client = CogniteClient() space_id = "scada-alarms" container_id = "alarm_events" mutable_stream_id = "alarm_data_live_stream" immutable_stream_id = "alarm_data_archive_stream" container = RecordContainerId(space=space_id, external_id=container_id) # Query mutable stream for distribution mutable_stream_results = client.data_modeling.records.aggregate( stream_id=mutable_stream_id, last_updated_time=TimeRange( gt=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%S.00Z"), ), filter=HasData(containers=[container]), aggregates={ "total_count": Count(), "by_acknowledged": UniqueValues(property=[space_id, container_id, "is_acknowledged"]), }, ) print("MUTABLE Stream Statistics:") print(f" Total Records: {mutable_stream_results['total_count'].value}") for bucket in mutable_stream_results["by_acknowledged"].buckets: is_ack = bucket.value == "true" if isinstance(bucket.value, str) else bucket.value ack_status = "Acknowledged (True)" if is_ack else "Unacknowledged (False)" print(f" {ack_status}: {bucket.count} records") # Query immutable stream for distribution immutable_stream_results = client.data_modeling.records.aggregate( stream_id=immutable_stream_id, last_updated_time=TimeRange( gt=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%S.00Z"), ), filter=HasData(containers=[container]), aggregates={ "total_count": Count(), "by_acknowledged": UniqueValues(property=[space_id, container_id, "is_acknowledged"]), }, ) print("\nIMMUTABLE Stream Statistics:") print(f" Total Records: {immutable_stream_results['total_count'].value}") for bucket in immutable_stream_results["by_acknowledged"].buckets: is_ack = bucket.value == "true" if isinstance(bucket.value, str) else bucket.value ack_status = "Acknowledged (True)" if is_ack else "Unacknowledged (False)" print(f" {ack_status}: {bucket.count} records") ``` ```javascript JavaScript/TypeScript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { CogniteClient } from '@cognite/sdk'; const client = new CogniteClient({ appId: 'records-tutorial' }); const spaceId = 'scada-alarms'; const containerId = 'alarm_events'; const mutableStreamId = 'alarm_data_live_stream'; const immutableStreamId = 'alarm_data_archive_stream'; const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); // Query mutable stream for distribution console.log('MUTABLE Stream Statistics:'); const mutableAggResults = await client.records.aggregate(mutableStreamId, { lastUpdatedTime: { gt: oneDayAgo, }, filter: { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, aggregates: { total_count: { count: {} }, by_acknowledged: { uniqueValues: { property: [spaceId, containerId, 'is_acknowledged'], }, }, }, }); console.log(` Total Records: ${mutableAggResults.total_count.count}`); const mutableBuckets = mutableAggResults.by_acknowledged.uniqueValueBuckets; mutableBuckets.forEach((bucket) => { const isAck = bucket.value === 'true' || bucket.value === true; const ackStatus = isAck ? 'Acknowledged (True)' : 'Unacknowledged (False)'; console.log(` ${ackStatus}: ${bucket.count} records`); }); // Query immutable stream for distribution console.log('\nIMMUTABLE Stream Statistics:'); const immutableAggResults = await client.records.aggregate(immutableStreamId, { lastUpdatedTime: { gt: oneDayAgo, }, filter: { hasData: [ { type: 'container' as const, space: spaceId, externalId: containerId, }, ], }, aggregates: { total_count: { count: {} }, by_acknowledged: { uniqueValues: { property: [spaceId, containerId, 'is_acknowledged'], }, }, }, }); console.log(` Total Records: ${immutableAggResults.total_count.count}`); const immutableBuckets = immutableAggResults.by_acknowledged.uniqueValueBuckets; immutableBuckets.forEach((bucket) => { const isAck = bucket.value === 'true' || bucket.value === true; const ackStatus = isAck ? 'Acknowledged (True)' : 'Unacknowledged (False)'; console.log(` ${ackStatus}: ${bucket.count} records`); }); ``` **Expected output:** ``` MUTABLE Stream Statistics: Total Records: 38 Unacknowledged (False): 38 records IMMUTABLE Stream Statistics: Total Records: 64 Acknowledged (True): 64 records ``` This confirms that all 64 acknowledged alarms have been successfully moved from the mutable stream (which now contains only the remaining 38 unacknowledged alarms) to the immutable archive stream. Together, this accounts for the original 102 total records (38 + 64 = 102). You've successfully built a complete SCADA alarm management pipeline using Records! You've created schemas, set up mutable and immutable streams, ingested alarm data, updated records, queried them using multiple endpoints, and implemented a stream-to-stream archiving pattern. This pipeline demonstrates the full lifecycle of managing high-volume OT event data with Records. ## Advanced: Build a stream-to-stream transformation pipeline A powerful architectural pattern for Records is creating data pipelines that move and transform data between streams. This separates raw data ingestion from creating clean, standardized datasets for consumption. You'll build a three-stage pipeline: ingest raw data to a source stream, transform it using a Hosted REST Extractor, and consume the standardized data from a destination stream. To build a stream-to-stream transformation pipeline, you need to configure these components: 1. **Source**: Connects to the CDF API to read from the source stream's `/sync` endpoint. 2. **Destination**: Provides credentials to write transformed records back to CDF. 3. **Mapping**: Transforms records from the source stream format to the destination stream format. 4. **Job**: Ties everything together and runs on a schedule with pagination and incremental load handling. The source defines how the REST extractor connects to the CDF API to read records from your source stream. Since you're reading from CDF, use your CDF cluster as the host and configure OAuth client credentials authentication. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.hosted_extractors.sources import ( RestSourceWrite, RESTClientCredentialsAuthenticationWrite ) source = client.hosted_extractors.sources.create( RestSourceWrite( external_id="my-records-source", host=".cognitedata.com", scheme="https", authentication=RESTClientCredentialsAuthenticationWrite( client_id="", client_secret="", token_url="", scopes="" ) ) ) ``` Replace ``, ``, ``, ``, and `` with your actual values. The source credentials must have `streamrecords:READ` capability scoped to the source stream's space. The destination provides credentials for the extractor to write transformed records to CDF. Create a session and use its nonce to authenticate the destination. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.hosted_extractors import DestinationWrite, SessionWrite from cognite.client.data_classes.iam import ClientCredentials # Create a session for the destination session = client.iam.sessions.create( client_credentials=ClientCredentials( client_id="", client_secret="", ), session_type="CLIENT_CREDENTIALS" ) # Create the destination using the session nonce destination = client.hosted_extractors.destinations.create( DestinationWrite( external_id="my-records-destination", credentials=SessionWrite(session.nonce) ) ) ``` The destination credentials must have `streamrecords:WRITE` capability scoped to the destination stream's space. The mapping defines how to transform each record from the source stream into the output format for the destination stream. The `/sync` endpoint returns a response containing an `items` array. The mapping iterates over these items and transforms each one. For writing to Records streams, the output must include: * `type`: Set to `"immutable_record"` for immutable streams. * `space`: The space where the destination stream exists. * `stream`: The external ID of the destination stream. * `externalId`: A unique identifier for each record. * `sources`: An array of container sources with their properties. * `beta`: Set to `true` while the records destination behavior is still in preview (the JSON field name remains `beta`). See [release states and availability](/cdf/product_feature_status). ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.hosted_extractors.mappings import MappingWrite, CustomMapping mapping_expression = """ input.items.map(item => { "type": "immutable_record", "space": "my-space", "externalId": item.externalId, "stream": "my-destination-stream", "beta": true, "sources": [ { "source": { "space": "my-space", "externalId": "my-container", "type": "container" }, "properties": item.properties["my-space"]["my-container"] } ] }) """ mapping = client.hosted_extractors.mappings.create( MappingWrite( external_id="my-records-mapping", mapping=CustomMapping(mapping_expression), published=True, input="json" ) ) ``` See [custom mappings](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings) for the full mapping language reference. The job ties everything together. It reads from the source stream's `/sync` endpoint on a schedule, applies the mapping, and writes to the destination. Configure **incremental load** to initialize the cursor on the first run, and **pagination** to handle cursor-based iteration through records. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.hosted_extractors.jobs import ( JobWrite, CustomFormat, RestConfig ) from cognite.client.data_classes.hosted_extractors import BodyLoad source_stream_id = "my-source-stream" project = client.config.project job = client.hosted_extractors.jobs.create( JobWrite( external_id="my-records-sync-job", destination_id="my-records-destination", source_id="my-records-source", format=CustomFormat(mapping_id="my-records-mapping"), config=RestConfig( interval="1h", path=f"/api/v1/projects/{project}/streams/{source_stream_id}/records/sync", method="post", headers={"Content-Type": "application/json"}, body={ "limit": 1000, "filter": { "hasData": [ { "type": "container", "space": "my-space", "externalId": "my-container" } ] } }, # Initialize cursor on first run incremental_load=BodyLoad( value='{...context.request.body, ...{"initializeCursor": "30d-ago"}}' ), # Use cursor from response for subsequent requests pagination=BodyLoad( value='if(body.nextCursor, {...context.request.body, ...{"cursor": body.nextCursor}}, null)' ), ), ) ) ``` The job configuration includes: | Parameter | Description | | - | - | | **interval** | How often the job runs (for example, `1h`, `15m`, `1d`). | | **path** | The `/sync` endpoint path for the source stream. | | **body** | The base request body with limit and filter parameters. | | **incremental\_load** | Sets `initializeCursor` on the first request to start syncing from a specific point (for example, `30d-ago`). | | **pagination** | Uses `nextCursor` from the response to paginate through all records. Returns `null` to stop when no more records exist. | The `incremental_load` expression merges the base body with the `initializeCursor` parameter. The `pagination` expression checks if `nextCursor` exists in the response and either continues with the cursor or returns `null` to stop pagination. After creating the job, you can monitor its status, logs, and metrics. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Check job logs logs = client.hosted_extractors.jobs.list_logs(job="my-records-sync-job") # Check job metrics metrics = client.hosted_extractors.jobs.list_metrics(job="my-records-sync-job") ``` **Expected outcome**: The job runs on the configured interval, reads records from the source stream, transforms them using your mapping, and writes them to the destination stream. This composable architecture lets you ingest data once in its raw format, then create streams containing standardized, use-case-specific record schemas from the source stream. This builds a robust and maintainable data processing ecosystem within CDF for your high-volume OT event and log data. # Agents Source: https://docs.cognite.com/cdf/explore/agents Work with Atlas AI agents trained on your data and customized for your organization's specific use cases. Your organization develops and maintains the agents, and you need to contact your local admin for help. If you don't see any agents on the Agents page, there are no Atlas AI agents available within your organization, or you don't have the necessary capabilities to access the agents. Atlas AI features require a separate license from your standard Cognite Data Fusion (CDF) subscription. To enable Atlas AI functionality for your organization, contact your Cognite representative or fill out [this form](https://www.cognite.com/en/contact). **Learn more:** [About Atlas AI](/cdf/atlas_ai/). # Canvas Source: https://docs.cognite.com/cdf/explore/canvas Step-by-step guide to create a canvas, upload files, add time series and diagrams, work with PDFs, build cause maps, organize canvases with labels, and use agents to analyze industrial data. Use Canvas in Cognite Data Fusion (CDF) to investigate industrial data visually. Create a canvas, upload files, and add project data such as time series and engineering diagrams. You can also work with PDFs, build cause maps, organize your canvases with labels, and use agents during analysis. ## Create a canvas Navigate to Flows > Canvas. Select + Create canvas to start a new canvas. Drag a [supported file or image](#supported-file-formats) from your device onto the canvas or select + Add data to find and select data from CDF. Files you upload are visible to all users with access to that canvas. Canvas workspace with toolbar, settings, menubar, agents, and Add data callouts ➀ **Toolbar** — hover over each tool to see a description. ➁ **Settings** — select canvas **settings**, add or remove **[labels](#labels)**, **[save as PDF](#download-files)**, **lock**, or **duplicate** a canvas. ➂ **Menubar** — select an item on the canvas to show **relevant options** for the item, for example, [Explore with AI](/cdf/flows_workspace/flows_workspace#summarize-or-ask-questions-about-documents). ➃ **Agents** — open the [**Atlas** chat](#use-agent) and select an AI agent to work with the canvas. ➄ Add data — find and **select data** to add to the canvas. ## Navigate To navigate on the canvas: * To **pan**, use the Spacebar and the mouse, or use two or more fingers on a touchpad. * To **move** items, select and drag. * To **zoom**, use the mouse scroll wheel, or pinch and stretch on a touchpad. ## Labels Labels are keywords that describe the content on a canvas. Use them to organize your canvases and to find the right canvas faster. ### Assign or remove labels On a canvas, select the arrow next to the canvas name > Edit name and labels, and then select the labels you want. You can choose from the labels an administrator has created. ### Filter on labels On the canvas overview, where all canvases are listed, filter on labels to narrow the list. To create labels and set up the capabilities required to manage them, see [Canvas labels](/cdf/configure/canvas_settings). ## Use an agent to find data Open an agent on the canvas to find data from selected locations and add it to your investigation. Agent panel on a canvas with session, location, Add to Canvas, and agent selector callouts ➀ **New session** — start a new conversation with an agent. ➁ **Select locations** — see your current location. Select locations to choose which data the agent can access. ➂ **Add to Canvas** — add the selected data to the canvas. ➃ **Select agent** — select the agent you want to start a conversation with. To interact with an agent, you need these [capabilities](/cdf/access/guides/capabilities#atlas-ai). Learn more about [Atlas AI agents](/cdf/atlas_ai/concepts). ## Engineering diagrams Add engineering diagrams and related data to a canvas. Engineering diagram on a canvas with related data and live time series callouts ➀ **Add related data** — select a purple or orange link and the plus sign (+) to add related data, such as engineering diagrams, files, and time series. ➁ **View live time series data** — pin a time series to the canvas, set threshold values, and use colors to visualize any breaches. ## Time series Add time series to a canvas from Add data, or from related data on an engineering diagram. ### Adjust time series display After you add a time series to a canvas, you can switch to value view, turn off auto-refresh to see historical data, and set how many digits appear after the decimal point. Select the time series on the canvas, then select Change to value view. The time series switches to value view, and additional tools appear on the menubar. Auto-refresh keeps the values updated and synchronized every 5 minutes as they change in CDF. To stop the values from updating automatically, select Disable auto refresh. With auto-refresh off, you can select a date in the past to see historical data for the time series. Select Adjust fractional digits (.5x), then use the up and down arrows to set how many digits appear after the decimal point. The minimum is 1 (`.0`) and the maximum is 3 (`.000`). ## Cause maps Use a cause map on a canvas to structure a root-cause analysis or a fault-tree investigation. Build a tree of cause nodes under the root node. As you add nodes, the tree auto-aligns. Add a cause map from the canvas toolbar. A root node appears on the canvas. Select (+) on a node to add a child cause node. When you add a node, choose the direction it should grow, horizontal or vertical. You can't switch that orientation later. Continue selecting **+** on nodes to grow the tree. The layout auto-aligns as you add nodes. Drag the root node to move the entire cause map tree on the canvas. ## Work with PDF files When you add a PDF file to a canvas, you can view it directly, extract individual pages for comparison, and download the canvas or specific pages as a PDF. ### Extract pages Extracting pages lets you view multiple pages side by side for comparison or analysis. Select a file and, on the toolbar, select **Extract pages** . Select all pages, specific pages, or a range of pages to place them on the canvas. Extract pages doesn't create a new file. Each extracted page is a view of the original file focused on a specific page. You can use the page controls to navigate to other pages from any extracted view. For example, if you extract pages 3-6 from a 10-page PDF, you get 4 additional views of the original file, each focused on a specific page from 3 to 6. If you extract all 10 pages, you get 10 additional views next to the original file. ### Download files **Download a canvas** Use **Download as PDF** to save the canvas and its contents as a PDF file. Select the canvas name > **Download as PDF**, select the containers and files to include, and select **Download**. By default, all containers and files on the canvas are selected. Clear the checkboxes for any items you don't want to include. You can also reorder the pages before downloading. **Download a file** Download an individual file directly from the toolbar. Select the file and then select **Find related data** > **Download**. **Download specific pages from a PDF** If you only need certain pages from a PDF file, select the canvas name > **Download as PDF**, select the checkbox next to the file, and select **Multiple pages**. Switch to **Selection**, specify the pages you need, and select **Download**. ## Supported file formats You can upload these file types to a canvas. | MIME type | Extensions | Name | | :- | :- | :- | | `application/json` | `.json` | JavaScript Object Notation | | `application/msword` | `.doc` | Microsoft Word document (older format) | | `application/pdf` | `.pdf` | Portable Document Format | | `application/rtf` | `.rtf` | Rich Text Format | | `application/vnd.apple.numbers` | `.numbers` | Apple Numbers spreadsheet | | `application/vnd.ms-excel` | `.xls` | Microsoft Excel spreadsheet (older format) | | `application/vnd.ms-excel.sheet.binary.macroEnabled.12` | `.xlsb` | Microsoft Excel Binary Workbook | | `application/vnd.ms-excel.sheet.macroEnabled.12` | `.xlsm` | Microsoft Excel Macro-Enabled Workbook | | `application/vnd.ms-excel.template.macroEnabled.12` | `.xltm` | Microsoft Excel Macro-Enabled Template | | `application/vnd.ms-powerpoint` | `.ppt` | Microsoft PowerPoint presentation (older format) | | `application/vnd.ms-powerpoint.presentation.macroEnabled.12` | `.pptm` | Microsoft PowerPoint Macro-Enabled Presentation | | `application/vnd.ms-powerpoint.slideshow.macroEnabled.12` | `.ppsm` | Microsoft PowerPoint Macro-Enabled Slide Show | | `application/vnd.ms-powerpoint.template.macroEnabled.12` | `.potm` | Microsoft PowerPoint Macro-Enabled Template | | `application/vnd.ms-word.document.macroEnabled.12` | `.docm` | Microsoft Word Macro-Enabled Document | | `application/vnd.ms-word.template.macroEnabled.12` | `.dotm` | Microsoft Word Macro-Enabled Template | | `application/vnd.oasis.opendocument.presentation` | `.odp` | OpenDocument Presentation | | `application/vnd.oasis.opendocument.presentation-template` | `.otp` | OpenDocument Presentation template | | `application/vnd.oasis.opendocument.spreadsheet` | `.ods` | OpenDocument Spreadsheet | | `application/vnd.oasis.opendocument.text` | `.odt` | OpenDocument Text document | | `application/vnd.oasis.opendocument.text-template` | `.ott` | OpenDocument Text document template | | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | `.pptx` | Microsoft PowerPoint Open XML presentation | | `application/vnd.openxmlformats-officedocument.presentationml.slideshow` | `.ppsx` | Microsoft PowerPoint Open XML slideshow | | `application/vnd.openxmlformats-officedocument.presentationml.template` | `.potx` | Microsoft PowerPoint Open XML presentation template | | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | `.xlsx` | Microsoft Excel Open XML spreadsheet | | `application/vnd.openxmlformats-officedocument.spreadsheetml.template` | `.xltx` | Microsoft Excel Open XML spreadsheet template | | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | `.docx` | Microsoft Word Open XML document | | `application/vnd.openxmlformats-officedocument.wordprocessingml.template` | `.dotx` | Microsoft Word Open XML document template | | `image/jpeg` | `.jpg`, `.jpeg` | Joint Photographic Experts Group image | | `image/png` | `.png` | Portable Network Graphics image | | `image/svg+xml` | `.svg` | Scalable Vector Graphics image | | `image/tiff` | `.tif`, `.tiff` | Tagged Image File Format | | `image/webp` | `.webp` | Web Picture format | | `text/csv` | `.csv` | Comma-Separated Values | | `text/plain` | `.txt` | Plain text document | | `text/tab-separated-values` | `.tsv` | Tab-Separated Values | ## Further reading * [Search](/cdf/explore/search) — Find files and other industrial data in your project * [About Atlas AI agents](/cdf/atlas_ai/concepts) — Learn how agents work with industrial data * [Canvas capabilities](/cdf/access/guides/capabilities#canvas) — Access required to set up and use Canvas # Charts Source: https://docs.cognite.com/cdf/explore/charts Analyze time series data, create calculations, set thresholds, monitor alerts, assess data quality, and visualize activities. ## Create a chart Navigate to Flows > Charts. Select + New chart to start a new chart. To build on an existing chart, select ⋮ to the right of the chart > Duplicate. Charts toolbars ➀ Toolbar — hover over each tool to see a description. Use the Slider to view or compare time series values. ➁ Open in... — download, duplicate, delete, or Open the chart in Canvas. ➂ Add data — add a time series, or import or create a calculation. ➃ Agents — open an AI agent to work with the chart. ➄ Sidebar — set up alerting, monitoring, thresholds, or data profiling. Add events and activities. ### Use an agent to find and analyze data Atlas in Charts ➀ New session — start a new conversation with an agent. ➁ Select locations — see your current location. Select locations to choose which data the agent can access. ➂ Select agent — select the agent you want to start a conversation with. To interact with an agent, you need these [capabilities](/cdf/access/guides/capabilities#atlas-ai). Learn more about [Atlas AI](/cdf/atlas_ai/concepts). ### Add a time series To add a time series to your chart: Select Add data > Add time series. Find and select the time series > Add. For each time series and calculation, explore the different ways to set units and fine-tune the data display. For example, try to adjust the thickness and color of the diagram line. Click for more details. ### Add a calculation To add a calculation: Select Add data > Create calculation. Select Add node or right-click in the panel below the chart to add input sources. Charts calculations Right-click the panel to add **constants** or **functions** to the calculation. Select to view the function details or hover over the parameters for more information. Pipe the calculations in order and include the final **output** node to display the result on the chart plot. Use one calculation as input to another to view the output from each step. ### Schedule a calculation Schedule a calculation and save the result as a time series. You can also [monitor](#monitor-time-series) the calculation. [Add a calculation](#add-a-calculation) and select Save and Schedule. Specify the sign-in credentials for the schedule, and click Next. If you don't know the credentials, contact your CDF admin. Give the schedule a name and specify the necessary settings. Select Start schedule. ### Trend data To stack, overlay, and compare trends for time series, click+drag or hover+scroll to adjust and move each diagram axis. Use the Slider on the toolbar to view or compare time series values at a specific time. ### Set thresholds for historical data To set thresholds for historical data: On the top right of the chart, select the Threshold icon. In the panel, set the threshold values for time series and calculations. ### Add events You can add events, such as a faulty shutdown, to help analyze time series. To add events to a chart: Select Events from the sidebar. Add or select a filter to specify the events to add. ### Add activities Activities in Cognite Data Fusion (CDF) represent tasks, events, or operations that happen over a period of time in your industrial environment. They provide a structured way to track work, maintenance, inspections, and other time-based activities, connecting planned and executed work with your physical assets and operational data. Turn on Activity to visualize and contextualize time series data with these operational events in your chart. Activities appear as vertical bars in the chart, providing visual reference to help you correlate time series behavior with relevant operational activities such as maintenance work, inspections, or production events. #### Prerequisites Before you add activities to a chart, ensure that: * The relevant data is modeled as [activities or extended activities in your CDF project](/cdf/dm/dm_reference/dm_core_data_model#cogniteactivity). * You have time series `read` access rights to view Charts and the relevant data. To add activities to a chart: Select + Add data > Add time series. Search and select the time series you want to visualize. In the Category dropdown, select Activity, and then select the activities you want to visualize. Once selected, the Activity layer will turn on automatically. The Asset filter applies automatically based on the time series the assets belong to in the chart. You can clear or add filters based on your preferences. Activities appear as vertical blue bars on the Activity layer. These bars help you visually correlate time series with relevant activities. Activities that start before or end after the selected time range are also shown, so you don't miss activities that overlap the edges of your view. The bars have varying levels of transparency to distinguish overlapping activities and keep multi-layer views readable. To work with activities: Click an activity bar to view details such as start and end times, a description, and the asset the activity is tied to. Click View details to see full information about the activity. The date range on the Activity page matches the time range set in your chart, including extended activities. Click the pin icon to keep the activity highlighted while you compare it with other activities. In the search panel, open the Activity category and hover over an activity to highlight the matching bar in the chart. This works for activities already added to the chart. If an activity is already pinned, it can appear shadowed while you hover a different activity. ## Monitor time series Time series monitoring jobs send [emails for any breaches to the thresholds](/cdf/configure/charts_monitoring) you set. If there are several breaches within the *minimum duration period*, you'll only receive one email for the period. We recommend that you set a minimum duration of 5 minutes to limit the number of email alerts you receive. The minimum expected data point frequency is 1 minute. If the data points have a lower frequency, alerts may not trigger as expected. ### Create monitoring jobs Select Monitoring > Create from the sidebar. Give the monitoring job a **name**. Select the time series or calculation you want to monitor in the Source dropdown. Specify the **threshold** and **minimum duration** for the alert. Specify the folder to store the alerts in. Select the users who'll receive the email alerts. Open Advanced options and specify the sign-in credentials for the monitoring job. Select Start monitoring. ### Subscribe to alerts To **subscribe** to an alert: Select Monitoring from the sidebar. Find the alert you want to subscribe to and select . ### Resolve alerts To **resolve** an alert you've investigated: Select the alert > History. Select Active > Mark as resolved. ## Data aggregation If a chart contains more than 100,000 data points, Charts summarizes the data points to make the data easier to view. The aggregated data is shaded and shows the minimum and maximum values of the aggregated data range. Zoom in to view individual data points. # Custom apps Source: https://docs.cognite.com/cdf/explore/custom_apps Access custom applications built for your organization's specific workflows. Custom apps are specialized applications that your organization develops and maintains to support specific industrial workflows and business processes. These applications appear on the Custom apps page in the Flows workspace, where you can discover and access tools tailored to your organization's unique needs. ## How to find and access apps Navigate to the Flows workspace in Cognite Data Fusion. Select Custom apps from the sidebar or main navigation. Use the filtering and sorting options to find the applications you need. If you don't see any applications on the Custom apps page, this means: * There are no applications available within your organization, or * You don't have the necessary capabilities to access the apps Your organization develops and maintains these applications, and you need to contact your local admin for help. ## Further reading * [Flows custom apps](/cdf/flows) — Create production-ready React applications with AI-native development tools * [Streamlit custom apps](/cdf/streamlit) — Build data visualization tools and dashboards using Python # Browse 3D Source: https://docs.cognite.com/cdf/explore/navigate_3d Explore physical assets in 3D environments and search for equipment within location models. ## Open the 3D view Select Browse 3D from the Home page, or go to Search > 3D view. You can also open scenes, 360° image collections, CAD models, and point clouds directly from the **3D preview** in the right-hand sidebar of [Search](/cdf/explore/search). If you don't see any models in Browse 3D, there are no 3D models available within your organization. ## Navigate, analyze, and search in 3D The 3D view provides tools for exploring and analyzing your 3D environment. Browse 3D view ➀ **3D toolbar** — customize your view, analyze and measure objects, and adjust visualization options. * **Tree view** : view and filter models and other elements in the scene. * **Cropping tool** : create a slice plane or crop box to define a custom 3D boundary. * **Measurements** : measure point coordinates, distances, areas, volumes, and diameters. * **Points of interest** : bookmark specific views and asset selections. * **Minimap** : track your location and navigate in large scenes. To configure the minimap, see [Configure a scene](/cdf/3d/guides/scene). * **Ghost mode** : focus on specific parts while fading non-selected elements. ➁ **Scene dropdown** — navigate to other 3D models and scenes available for your location. ➂ **Top bar** — change viewing perspectives, adjust camera speed, and focus on specific elements within your models. ➃ **Search panel** — search for an asset directly in the 3D view and select Open for further analysis in [Search](/cdf/explore/search). # Search Source: https://docs.cognite.com/cdf/explore/search Step-by-step guide to finding industrial data in Cognite Data Fusion (CDF), from filtering and sorting results to previewing an item and downloading lists as CSV. Search helps you find industrial data across your organization in CDF, including assets, time series, files, events, and engineering diagrams. Preview a result to see its properties and related data, then open it in Charts or Canvas for further analysis. ## Start a search In the Flows workspace, navigate to the Home page or the Search tool: Click the Search field, type what you're looking for, and press Enter. Search returns data that matches your query precisely. For example, searching for `21-PT-10-19` returns results that contain all four parts (`21`, `PT`, `10`, `19`). If you find a pump, you'll also find associated assets, documents, sensor data, and engineering diagrams. Use [filters](#filter-and-sort-your-search-results) to refine your search results further. Select a result to [preview](#preview-your-search-results) more details and open in Charts or Canvas for further analysis. ## Filter and sort your search results To help you find exactly what you need, you can filter and sort your search results. Filter and sort search results ➀ **Categories** — filter the results by data **categories**, for example, to see only matching time series. ➁ **Location** — select the **locations** you want to see results from. ➂ **Filters** — select one or more Filters to further refine the results for a category. ➃ **Sort** — click any of the **column headings** to sort the results. ➄ **3D** — open a 3D model to explore physical assets, such as rigs or locations. Refer to [Browse 3D](/cdf/explore/navigate_3d) for more information. ## Preview your search results Select a result to preview the details in the right-hand sidebar, and open the item in Charts or Canvas for further analysis. Preview search results ➀ **Overview** — see an **overview of the information** about a search result. ➁ **Details tabs** — view **detailed information** about the search result. The tabs vary by category. For example, Tree view is only available for hierarchical data like *assets*. ➂ **Open** — view the item in full screen and close the sidebar. ➃ **Open in Charts/Canvas** — open the item in Charts or Canvas for further analysis, or **download** a file. ## Understanding categories and tree view When you search for assets, you'll notice different ways to organize and view your results. ### Categories help you focus your search **Categories** group similar types of equipment or assets together. When you select a category like "Pumps" or "Sensors", you'll see only items of that type in your results. ### Tree view shows how assets connect **Tree view** displays how your assets are organized hierarchically, for example, showing that a pump belongs to a specific system, which belongs to a plant location. Search results and Tree view show the full asset hierarchy even when it's split across different categories. Items from other categories appear **grayed out** so you still have full context. For example, if you're viewing "Pumps" but a pump's parent system is in the "Systems" category, that parent appears grayed out. The grayed-out items are still clickable and interactive. If you don't see the full asset hierarchy you expect, try selecting a different **Category** or adjusting your **Location** filters. You can also ask your administrator about which asset types are available for search. ## Download instances and relationship lists You can download lists of data instances from search results and from related data sections on the instance page as a CSV file. All CSV files include audit information: who downloaded, when, and a source URL that links back to the search page in CDF. Anyone with permission to view those search results can follow the link. Downloads from related data sections also include the properties (metadata) of the instance that owns the relationship. * **Files**: Only file metadata is exported, not the file itself. * **Time series**: Data points are not exported. Both search results and downloads are limited to 1,000 instances due to an API limit, even if the total number of matches exceeds that. ### Download instances Enter your search term in the Search field and press Enter. Then use [filters](#filter-and-sort-your-search-results), sorting, and column selection to refine your results. Click the download button to download the search results as a CSV file. ### Download relationship lists Select a related data section on the instance page, for example, **Assets** > **Path**. Click the download button to download the relationship list as a CSV file. ## Upload files To upload files directly in Search, an admin must configure the space for file uploads. Go to **Admin** > **Project settings** > **Locations** > **Configure** > **Space for file upload**, set the space for file uploads. Only one space can be set per location. Upload is only available for data model views. Once the space is configured, you can upload files directly in Search. Files may take a moment to appear after uploading. ## Work with PDF files You can find information in PDF files using AI document analysis and text search. PDF file open on the Preview tab with the Explore with AI and Find text options On the Preview tab, you can: ➀ **[Explore with AI](/cdf/flows_workspace/flows_workspace#summarize-or-ask-questions-about-documents)** — use AI to summarize and ask questions about the PDF file. ➁ **Find text** — search for a specific word within the PDF file. Matches are highlighted, and you can browse through the results. ## Not finding what you're looking for? Check your spelling and make sure you've selected the correct [location](#filter-and-sort-your-search-results). You'll only see items that you have access to. Verify with your internal help desk or the CDF admin that you have access to the data you're looking for. ## Supported file formats Documents **Preview** supports the following file formats: | MIME type | Extensions | Name | | :- | :- | :- | | `application/cdf` | `.cdr` | CorelDRAW Image | | `application/dicom` | `.dicom` | Digital Imaging and Communications in Medicine | | `application/epub+zip` | `.epub` | Electronic Publication | | `application/json` | `.json` | JavaScript Object Notation | | `application/msword` | `.doc`, `.dot` | Microsoft Word document (older format) | | `application/pdf` | `.pdf` | Portable Document Format | | `application/postscript` | `.eps`, `.ps` | PostScript | | `application/rtf` | `.rtf` | Rich Text Format | | `application/vnd.apple.numbers` | `.numbers` | Apple Numbers spreadsheet | | `application/vnd.hp-pcl` | `.pcl` | Printer Command Language | | `application/vnd.ms-excel` | `.xls`, `.xlt` | Microsoft Excel spreadsheet (older format) | | `application/vnd.ms-excel.sheet.binary.macroEnabled.12` | `.xlsb` | Microsoft Excel Binary Workbook | | `application/vnd.ms-excel.sheet.macroEnabled.12` | `.xlsm` | Microsoft Excel Macro-Enabled Workbook | | `application/vnd.ms-excel.template.macroEnabled.12` | `.xltm` | Microsoft Excel Macro-Enabled Template | | `application/vnd.ms-htmlhelp` | `.chm` | Microsoft Compiled HTML Help | | `application/vnd.ms-powerpoint` | `.pot`, `.pps`, `.ppt` | Microsoft PowerPoint presentation (older format) | | `application/vnd.ms-powerpoint.presentation.macroEnabled.12` | `.pptm` | Microsoft PowerPoint Macro-Enabled Presentation | | `application/vnd.ms-powerpoint.slideshow.macroEnabled.12` | `.ppsm` | Microsoft PowerPoint Macro-Enabled Slide Show | | `application/vnd.ms-powerpoint.template.macroEnabled.12` | `.potm` | Microsoft PowerPoint Macro-Enabled Template | | `application/vnd.ms-word.document.macroEnabled.12` | `.docm` | Microsoft Word Macro-Enabled Document | | `application/vnd.ms-word.template.macroEnabled.12` | `.dotm` | Microsoft Word Macro-Enabled Template | | `application/vnd.ms-xpsdocument` | `.xps` | XML Paper Specification | | `application/vnd.oasis.opendocument.graphics` | `.odg` | OpenDocument Graphics | | `application/vnd.oasis.opendocument.graphics-template` | `.otg` | OpenDocument Graphics Template | | `application/vnd.oasis.opendocument.presentation` | `.odp` | OpenDocument Presentation | | `application/vnd.oasis.opendocument.presentation-template` | `.otp` | OpenDocument Presentation template | | `application/vnd.oasis.opendocument.spreadsheet` | `.ods` | OpenDocument Spreadsheet | | `application/vnd.oasis.opendocument.text` | `.odt` | OpenDocument Text document | | `application/vnd.oasis.opendocument.text-template` | `.ott` | OpenDocument Text document template | | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | `.pptx` | Microsoft PowerPoint Open XML presentation | | `application/vnd.openxmlformats-officedocument.presentationml.slideshow` | `.ppsx` | Microsoft PowerPoint Open XML slideshow | | `application/vnd.openxmlformats-officedocument.presentationml.template` | `.potx` | Microsoft PowerPoint Open XML presentation template | | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | `.xlsx` | Microsoft Excel Open XML spreadsheet | | `application/vnd.openxmlformats-officedocument.spreadsheetml.template` | `.xltx` | Microsoft Excel Open XML spreadsheet template | | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | `.docx` | Microsoft Word Open XML document | | `application/vnd.openxmlformats-officedocument.wordprocessingml.template` | `.dotx` | Microsoft Word Open XML document template | | `application/vnd.sun.xml.calc` | `.sxc` | StarOffice Calc Spreadsheet | | `application/x-mobipocket-ebook` | `.mobi` | Mobipocket eBook | | `application/x-ms-wmz` | `.wmz` | Windows Media Player Skin | | `application/x-tex` | `.tex` | TeX Source | | `application/xml` | `.xml` | XML Document | | `image/bmp` | `.bmp` | Bitmap Image | | `image/dib` | `.dib` | Device Independent Bitmap | | `image/dng` | `.dng` | Digital Negative | | `image/jpeg` | `.jpg`, `.jpeg` | Joint Photographic Experts Group image | | `image/png` | `.png` | Portable Network Graphics image | | `image/svg+xml` | `.svg`, `.svgz` | Scalable Vector Graphics image | | `image/tga` | `.tga` | Truevision TGA | | `image/tiff` | `.tif`, `.tiff` | Tagged Image File Format | | `image/vnd.djvu` | `.djvu` | DjVu Image | | `image/vnd.mozilla.apng` | `.apng` | Animated Portable Network Graphics | | `image/webp` | `.webp` | Web Picture format | | `message/rfc822` | `.mht`, `.mhtml` | MHTML Web Archive | | `text/csv` | `.csv` | Comma-Separated Values | | `text/html` | `.html` | HTML Document | | `text/markdown` | `.md` | Markdown Document | | `text/plain` | `.txt` | Plain text document | | `text/tab-separated-values` | `.tsv` | Tab-Separated Values | ## Further reading * [Charts](/cdf/explore/charts) — visualize and analyze time series data. * [Canvas](/cdf/explore/canvas) — collect and share data on a collaborative canvas. * [Browse 3D](/cdf/explore/navigate_3d) — explore physical assets in a 3D model. * [Flows workspace](/cdf/flows_workspace/flows_workspace) — work with AI document analysis and the other tools in the workspace. # Flows workspace Source: https://docs.cognite.com/cdf/flows_workspace/flows_workspace Navigate the Flows workspace sidebar to switch tools and projects, chat with Cognite Atlas, and manage help and profile settings. Start exploring by searching for data, or open a tool from the sidebar. You can also chat with Cognite Atlas from any Flows tool. ## Sidebar Use the sidebar to navigate the workspace. Flows workspace sidebar with Home, tools, and Ask Atlas AI ➀ Switch workspace — to switch to a different workspace, click the Flows workspace name and then a workspace from the menu. ➁ Collapse/expand — click the icon to collapse (**\<\<**) or expand (**>>**) the sidebar, for example, to free up extra space while you're viewing a chart. ➂ Switch project — to switch to a different project, click the project name and then a project from the menu. ➃ Home — go to the main workspace page to search or resume your activity. ➄ Tools — select an industrial tool to **find** or **analyze** data. ➅ Ask Atlas AI — open Cognite Atlas chat, which is available across the Flows workspace. Chat with the built-in Atlas agent, or choose one of your organization's agents. Ask questions from Search, Canvas, Charts, custom apps, and other tools without leaving the work you are doing. See [Chat with Cognite Atlas](#chat-with-atlas). To pick up where you left off, select a recent chart or canvas under Resume your activity. ### Help and profile settings Find more information and settings in the lower left part of the sidebar. Flows workspace help and profile settings ➀ Help and resources — find links to documentation, training, and Cognite Support. ➁ Settings — click your profile name to manage your profile and preferences. ## Chat with Cognite Atlas Open Ask Atlas AI from the sidebar to ask questions about your industrial data. Chat with the built-in Atlas agent, or choose one of your own. The chat experience is the same in every Flows tool. In the chat, you can: * **Mention a resource** by typing `@` to pin a specific asset, time series, file, document, or activity. A symbol shows the resource type. Mentions tell the agent which item you mean when names are similar. * **Open a citation** to preview the source, for example a document page, and jump to [Search](/cdf/explore/search) to verify it if needed. * **Use a bar chart, KPI tile, table, or Charts time series** in the reply. You can inspect the result in the conversation. * **Rate a reply** with thumbs up or down and an optional comment. Cognite uses this to improve quality. To use Atlas AI, you need these [capabilities](/cdf/access/guides/capabilities#atlas-ai). If your organization configured agents, open them from [Agents](/cdf/explore/agents). ## Summarize or ask questions about documents In some of the tools, you can use Explore with AI to summarize and ask questions about PDF files. Explore with AI works best with PDF files up to 1 MB (approximately 500 pages). Other file types may work but aren't officially supported. Select Explore with AI to: * **Summarize**: Generate a summary of the document's content. * **Ask questions**: Get answers about the document based on its content. You can ask questions about multiple PDF files in the following ways: * [Search](/cdf/explore/search): Ask questions about files directly or indirectly linked to the main file on the Overview tab under Files. * [Canvas](/cdf/explore/canvas): Select multiple documents before asking questions. * [InField](/cdf/infield): Specify that you want information from several or all files. ## Further reading * [Agents](/cdf/explore/agents) * [Atlas AI capabilities](/cdf/access/guides/capabilities#atlas-ai) # About Flows Source: https://docs.cognite.com/cdf/flows_workspace/index Use the Flows workspace to find and analyze data. ### Navigate the workspace Use the sidebar to navigate between tools, switch projects, and access help resources. ### Find and explore data Search and discover industrial data, assets, and engineering diagrams across your organization. Explore physical assets in 3D environments and search for equipment within location models. ### Analyze and investigate Visualize sensor data, apply calculations, and create template charts for troubleshooting. Create and use canvas to analyze and investigate industrial data visually. ### Get assistance and extend Access custom applications built for your organization's specific workflows using Streamlit, Flows, or other technologies. Work with Atlas AI agents trained on your data and customized for your organization's specific use cases. # About Cognite Data Fusion (CDF) Source: https://docs.cognite.com/cdf/index Cognite Data Fusion (CDF) is an industrial DataOps platform that streams your industrial data into a structured, flexible, and contextualized knowledge graph. With Cognite, **you** own your data. We use it only to provide agreed-upon services, handle it [**securely**](/cdf/trust), and comply with **privacy** and **legal regulations**. If you leave our services, we ensure you continue to own your data. Cognite Data Fusion (CDF) runs in the [**cloud**](/cdf/admin/clusters_regions) and has a modular design that lets you stream data from various sources, contextualize it, and make it available through dedicated workspaces, APIs, and SDKs. The CDF architecture ## Implementation steps The following steps provide a high-level overview of implementing CDF. For a complete implementation guide, see [Deploy CDF](/cdf/deploy/). Before integrating data, define your **data governance** policies. Appoint a **CDF admin** to work with IT to ensure CDF follows your organization's [**security**](/cdf/trust/security/security_intro) practices. Connect CDF to your identity provider (IdP) to [**manage access**](/cdf/access) using existing user identities. CDF provides out-of-the-box [data models](/cdf/dm/) to build a structured, flexible, contextualized **knowledge graph**. [Add data](/cdf/integration/) from your **IT**, **OT**, and **ET** sources into CDF, including industrial control systems, ERP systems, and 3D CAD models. **Extract** data using standard protocols like PostgreSQL and OPC-UA, then **transform** it in the CDF [**staging area**](/cdf/integration/guides/extraction/raw_explorer) to fit the CDF data model. Use automatic and interactive [**contextualization**](/cdf/integration/concepts/contextualization/) tools that combine AI, machine learning, and domain expertise to map resources from different source systems to each other. With contextualized data in your industrial knowledge graph, use the built-in [Flows](/cdf/flows_workspace/) and build powerful apps and [AI agents](/cdf/atlas_ai/) to meet your business needs. All data is available through our [**REST-based API**](/api-reference/concepts/20230101/api-description). Cognite provides **connectors** and [**SDKs**](/dev/sdks/) for common programming languages and [analytics tools](/cdf/data_science/guides/ds_tools), like Python, JavaScript, Spark, OData (Excel, Power BI), and Grafana. The [**Functions**](/cdf/functions/admin) service provides a scalable, secure way to host and run Python code. # Industrial Data Science Library Source: https://docs.cognite.com/cdf/indsl # Create templates and checklists Source: https://docs.cognite.com/cdf/infield/guides/create_templates Create templates, add tasks and schedules, and generate checklists for field workers to perform at regular intervals. Use the Templates tab to create and plan tasks that the field workers perform at regular intervals, such as weekly installation operator rounds. Templates page navigation ① **Action menu** — create templates, add tasks, change the template's status, assign and create checklists, and more (...). ② **Filter** — filter the templates by name, status, labels, and assignee. By default, you see Ready templates assigned to you. You can select **X** to remove filters. ③ **Templates list** — view the list of templates with details. ④ **Tasks list** — view and interact with the list of tasks in your template. ⑤ **Add task** — fill out your tasks with the required details to perform the work in the field. ## Step 1. Create templates Select + Create template and name the template. Add existing or create new labels to organize, categorize, and filter work.

You can also add or remove labels on existing *Draft* templates by selecting … > **Edit template**. You can't delete labels from the system, only remove them from templates. To remove a label, select on the label. Labels added to templates are automatically inherited by all checklists created from these templates, including the individual tasks within those checklists.
Enter name and description, and, optionally, link the task to an asset. A description can be a step-by-step instruction for the task with links where necessary. Select Add to task and then select: Ask field workers to add measurement readings from equipment. You can set the measurement unit, for example, *bar* for pressure readings and minimum and maximum values. You can search and connect time series created in InField only. InField stores the measurements as time series in CDF, and you can trend and analyze the values using [Charts](/cdf/explore/charts). If you've **removed a numerical reading** from a checklist, it's still available as a time series. To create a numerical reading: 1. Select **+** and fill out the required fields to create the time series. 2. Optional. Link the time series to a specific asset, otherwise, the time series is linked to the root asset. 3. Optional. Enter the description for the time series. If provided, the description is used as the numerical reading name in checklists. Add to the task an item that requires a check, verification, inspection, etc. 1. Add a description of what needs to be done. 2. Enter the button name, select its color, and link the button to the task status. The task will change its status based on the button field workers select. For example: * When field workers work on the checklist and select the button with the linked **OK** status, their task status changes to **OK**. * If there's a numerical reading that's out of range (**Not OK**) and field workers select the button with the linked **OK** status, the task status will change to **Not OK** as this task requires attention. In this example, field workers should fill out all numerical readings and items to check for the status to change automatically. 3. Select + Add button to add another possible response to the description. **Schedule** tasks to automatically create them as checklists at regular intervals. You can set checklists to open automatically on specific days, every two weeks, twice a month on Mondays, every 2 years, etc. To schedule a task: 1. Set the time range when the field workers should complete the task. The initial start time is 6 AM in the computer's local time zone. You can change the start time to half-hour intervals. 2. Optional. Select the time zone . 3. Specify how often you want field workers to work on this task. For intervals other than daily, weekly, monthly, and yearly, set the start date when to create the first checklist from the task. By default, the date of adding a task is used. Once you schedule the task, hover over the calendar icon next to it to see when the next checklist is created. Ask field workers to give a written response to a request, question, message, etc. Set what subtask field workers need to perform if a checklist task gets a certain status. For example, if the task's status changes to Not OK, field workers need to explain the reason in free text. Select + Add task to add another task. One template has a limit of 500 tasks. However, to make it easy for the field worker to navigate them in the field, keep them as short as possible. Ready templates can be assigned to disciplines or users and can be [created as checklists manually and automatically](#step-3-create-checklists). Set the template to ready The templates left in Draft will remain templates and won't appear as checklists.
You need **extended access** to plan and schedule tasks, and you need to assign activities using the InField desktop version, not mobile. ## Step 2. Manage tasks To work with several tasks at once, you can: Select and manage tasks ① **Select tasks** — create task groups, schedule several tasks at once, duplicate, and delete tasks. ② **Edit a group** — select More options (…) to ungroup the tasks or rename the group. ## Step 3. Create checklists A checklist is a task or a list of tasks generated from a template or a work order. On the Templates tab, you can create a checklist from a template or its tasks in two ways: * **Manually**. Select a template in the Ready status and, on the right, select Create checklist. Suggest a start and end date and confirm the creation. The dates are used for sorting only. You'll create a checklist that's a copy of the template, even if some tasks are scheduled for a different date and time. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} classDiagram direction LR MyFirstTemplate --|> MyFirstChecklist : Create manually class MyFirstTemplate{ Task 1 - scheduled at 10am Task 2 - not scheduled Task 3 - not scheduled Task 4 - scheduled at 10am } class MyFirstChecklist{ Task 1 Task 2 Task 3 Task 4 } ``` Field workers can also create checklists on their [mobile devices](/cdf/infield/guides/perform#create-checklists). * **Automatically**. Once you add a schedule to a task or a group of tasks, the tasks automatically appear as checklists in the [Schedules](/cdf/infield/guides/plan#view-scheduled-work) tab. Remember to move the templates to Ready. The checklist created from the scheduled tasks excludes unscheduled template tasks. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} classDiagram direction LR MyFirstTemplate --|> MyFirstChecklist2 : Create automatically MyFirstTemplate --|> MyFirstChecklist3 : Create automatically class MyFirstTemplate{ Task 1 - scheduled at 10am Task 2 - scheduled at 6pm Task 3 - not scheduled Task 4 - scheduled at 6pm Task 5 - scheduled at 6pm Task 6 - scheduled at 10am } class MyFirstChecklist2{ Task 1 Task 6 } class MyFirstChecklist3{ Task 2 Task 4 Task 5 } ``` Automatic scheduling creates checklists with a delay unless you create them before the task start time. ## Delete and restore templates and tasks To delete a template, select More options (…) to the right of + Add task > Delete template. To restore a template, select Deleted in the left panel > More options (…) to the right of the template you want to restore > Restore template. You can't restore a task that has been deleted from a template. # Create and view observations Source: https://docs.cognite.com/cdf/infield/guides/observations Record equipment issues discovered during work, track their status, and send reports. Observations are information collected from field workers on malfunctioning equipment that needs further evaluation. Depending on whether you're on a desktop or mobile version, you can create and view or only create observations. The observations functionality may differ depending on your project configuration. On the Observations tab, you can: Observations page navigation ➀ **Create observations** — select + Create observation to fill out an observation form. ➁ **View observation details** — view the details of all observations and how many new observations you have today. Available on the desktop version only. ➂ **Select observations** — select observations to bulk delete or send to SAP (the SAP feature must be turned on). You can delete observations in the *Draft* and *Not sent* statuses, and send observations in the *Completed* status only. The SAP integration is only available for projects that are not using CDM (Core Data Model). An observation can have one of the following statuses: | Status | Meaning | | - | - | | Draft | The observation is in progress. | | Completed | The user filled out all information and selected *Completed*. | SAP-related statuses are: | Status | Meaning | | - | - | | Sent | The user completed the observation and sent it to SAP. | | File not sent | The observation was sent to SAP but not the attached media files. | | Not sent | The observation wasn't sent to SAP. | ## Create observations on mobile If you think an asset requires attention, you can create an observation for it: * When working on a checklist, select **(+)** > Create observation. * In the top left, select the menu tab > Create observation. # Perform activities Source: https://docs.cognite.com/cdf/infield/guides/perform Work with checklists to complete tasks, add measurement readings, record observations, and work offline on your mobile device. To perform activities in InField, you use checklists. Field workers can work on the checklists that are already added to the Checklists tab or create checklists from templates on the [Templates](#create-checklists) tab. A checklist has a set of tasks where you add a status, such as OK, Not OK, or Blocked. You can also add images, videos, and observations for malfunctions you notice while working. For some tasks, you may also see fields for adding measurement readings and check items. ## Work with checklists Checklists view on mobile ➀ **Filter checklists** — filter checklists by status, assignees, or use search. ➁ **Select a work week** — select a week to view checklists. By default, the view shows today at the top and the previous 6 days. To view upcoming work, select the next week to see checklists from tomorrow through the next 7 days. Select a checklist from the list. By default, you see Ready checklists assigned to you and your discipline. If you can't find a checklist, try sorting on the checklist status Ready, In progress, or Done, assignees, or use search. At the top of the checklist, select Group by and choose an asset property, for example, fire area. After you apply the grouping, you see the tasks in this checklist organized by that property. Use this to work through similar tasks together within the same checklist. Enter any measurement values your supervisor has requested. Once you start working on the checklist, the status changes from Ready to In progress. Respond to the supervisor's request by selecting an action button, for example, **Low**, **Normal**, **High**, or **Open**, **Closed**. For each task, select OK or a different status from Other. If you want to bulk change the statuses of the remaining tasks, select More options (…) > Set remaining tasks to and select the status. You can do that for the tasks in the group and outside the group and only once. Select + to add images and videos, enter text, or create observations. Fill out the [Observations](/cdf/infield/guides/observations) form if an asset requires a closer look, for example, due to malfunctions. When you've completed all tasks, set the checklist to Done. You can no longer update it. The checklist doesn't move automatically to Done when you complete all tasks. ## Create checklists To create checklists from templates on your mobile device: On the left panel, select Templates. You'll see the list of Ready templates assigned to you. Select to remove the filter. Select the template you want to create a checklist from and select Create checklist. The assignee is the person who created the checklist. Start working on the checklist or go back to create more checklists. ## Explore assets To see information linked to an asset, such as 3D images, documents, time series, or other assets, search for the asset or select an asset ID displayed as a blue link. * Select the search icon to search for assets, time series, or documents. * Select the search icon > Hierarchy to find equipment linked to the asset you're working with. ## Work offline Field workers can complete operator rounds and work orders with slow or no network connectivity on their mobile devices. To work offline, open the checklist while you still have network connectivity. If you go offline, you can still complete tasks but you can't access the asset explorer, upload media, or create observations. When you're back online, all data collected offline will be uploaded. To stop repetitive connectivity notifications, turn on airplane mode or ensure a steady network connection. To download data when offline, turn off airplane mode and wait until the data is uploaded. In the meantime, do NOT close the application or exit the checklist. For a minimum required bandwidth, refer to [Connectivity and bandwidth](/cdf/admin/connectivity). # Plan, schedule, and assign activities Source: https://docs.cognite.com/cdf/infield/guides/plan View work orders and checklists, assign activities to disciplines or users, create checklists from work orders, and track task progress. You can view and interact with the planned work on the Overview and the [Schedules](#view-scheduled-work) tabs. On the Overview tab, you'll see all planned activities divided into Work orders (maintenance work) and Checklists (operator rounds). You can assign these activities to a group of users (disciplines), such as all electricians, or directly to specific users. Also, you can suggest a start and end date for each activity, but these are only used for sorting. Overview page navigation On the Overview tab, you can: ➀ **Change location** — switch between different work locations. You can also select your profile at the top right > Manage account > Select location. ➁ **Search and filter** — adjust the table view with the search field, date picker, number of columns, and [save the view as a filter](#create-filters). ➂ **Create and assign** — [create checklists from a work order or its tasks](#create-checklists-from-work-orders) and assign or reassign work orders and checklists. ➃ **Change view** — switch between the Work orders and Checklists view. The Work orders tab shows the work orders imported from a work management application, such as SAP. Here, you can: * View work order details with a list of tasks (operations) to perform. Expand the task to see if it applies to several assets. * Select Create & Assign to create a checklist from work order tasks. Once you create the checklist, you can check its progress by selecting the number of tasks in the Checklist Progress column. * Create checklists from several work order tasks relevant to a field worker. If there are tasks left, you can also create another checklist. The work order will remain in progress until field workers complete all tasks in the work order. The **Checklists** tab shows recurring tasks generated from templates that are created by an admin on the Templates tab and planned checklists moved to the *Ready* state on the [Schedules](/cdf/infield/guides/plan#view-scheduled-work) tab. Here, you can: * View the checklist details. * Assign or reassign checklists. * [Track and complete checklist tasks](/cdf/infield/guides/plan#track-and-complete-tasks). * Delete checklists. If the field worker collects the data before you delete the checklist, the data is saved and stored in InField. You can only restore the deleted checklist through the Cognite API. When planning the work, you can also use Search to find and analyze time series, assets, and documents in your selected location. ## Create filters On the Overview tab, you can adjust the filters to view the most relevant data and save the selection as a filter. Create filters ➀ **Create** — save selected filters and create a filter preset you can use later. ➁ **Select** — select the saved filter preset to view relevant information. ➂ **Edit** — change saved filters, make them default, or delete them. To create a filter for columns: Select Hidden columns, select the columns to display, reorder them if needed, and select Apply. For example, select a certain checklist status. Next to Clear filters, select the dropdown arrow and save the filter. Name and save the filter. In the dropdown, you can see a list of all created filters. You can select a filter to apply to the Overview tab or make it default for the tab. You must create separate filters for work orders and checklists. Search results and a date range don't apply to the filter preset. ## Create checklists from work orders A checklist is a task or a list of tasks created from a [template](/cdf/infield/guides/create_templates) or created from a work order. You can create a checklist from a work order or its tasks (operations) only manually. Go to Overview > Work orders. Create a checklist from a work order or several work order tasks: * Select the work order to create a checklist from all tasks within it. * Expand the work order and select the operations you want on your checklist. A work order can consist of one or several operations. Operations are individual steps in an order or individual tasks in a checklist. Some operations that come from the work management system, such as SAP, have an order or an operation number. One operation in a work order can generate several tasks in a checklist if several operations have the same order/operation number and the same work order number (parentActivityId). ## Track and complete tasks As the field workers complete the tasks on a checklist, they update the status and add relevant images, measurements, and observations. You'll find the status of all activities along with their progress on the Overview tab, where you can sort the columns and select relevant dates. The table shows the meaning of the checklist status colors. | Color | Meaning | | - | - | | Grey | The operator hasn't completed any tasks. | | Green | The operator completed the checklist and all tasks have the OK or Not applicable state. | | Blue | The operator has started, but not completed, the checklist. | | Red | The operator set the state of at least one of the tasks to Not OK or Blocked. | You can update checklist statuses individually or in bulk: * **Individually** — select the checklist, then in the right panel select More options (…) > Set checklist to > select the status. * **In bulk** — select several checklists, then in the bottom panel select Set to Done. You can't update checklists with the status Done. However, **extended access** lets you change the status to In progress and update the checklist. ## View scheduled work The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. On the Schedules tab, you'll see all planned activities created from templates manually or automatically. As a template and checklist admin, you can view the checklists planned for the current and upcoming weeks and what has already been worked on, and move planned work to Ready. Schedules page navigation On the **Schedules** tab, you can: ➀ **Select a work week** — select the week to view details of checklists for past, current, and upcoming weeks. ➁ **View checklists** — review the checklist details, such as its status and assignees, who created it, and when the checklist becomes Ready. Here you can view Planned checklists. Planned is a transition status, meaning that the template is Ready and scheduled, but the checklist isn't Ready to start work on yet. Once the checklist reaches the scheduled date, it changes the status to Ready. ➂ **Move checklists to Ready** — manually transition checklists from Planned to Ready if you want field workers to start working on scheduled checklists before the planned date. ➃ **Preview checklists** — select a checklist to preview the task details before starting work. You can move checklists only in the Planned status. To move checklists to Ready: Select the week and day you want to view. Select the checkbox next to each checklist you want to move from Planned to Ready. To select multiple checklists, select the first checkbox, hold Shift, and select the last checkbox in the range. Select Move to Ready. Once moved, the checklists appear for field workers to start working on. You don't lose any data if the checklists aren't moved or are moved only partially. Repeat the process or select Retry to complete it. ## Template and checklist status flow Study the flowchart to see how template and checklist statuses interact and the key differences between them. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} flowchart TD subgraph Templates["Template Lifecycle"] T1[Draft Template] T2[Ready Template] T3[Deleted Template] T1 -->|Set to Ready| T2 T2 -->|Set to Draft| T1 T2 -->|Delete| T3 T1 -->|Delete| T3 T3 -->|Restore| T1 T3 -.->|Restore| T2 end subgraph Creation["Checklist Creation"] T2 -->|Create manually| C1[Ready Checklist] T2 -->|Scheduled task
reaches date/time| C2[Planned Checklist] C2 -->|Scheduled date
is reached| C1 C2 -->|Move to Ready
manually| C1 end subgraph Checklists["Checklist Lifecycle"] C1 -->|Field worker
starts work| C3[In Progress] C3 -->|Complete
all tasks| C4{Task States?} C4 -->|All OK or
Not applicable| C5[Done - Green] C4 -->|Any Not OK
or Blocked| C6[Done - Red] C1 -->|Delete| C7[Deleted Checklist] C3 -->|Delete| C7 C5 -->|Extended access:
Set to In Progress| C3 C6 -->|Extended access:
Set to In Progress| C3 end style T1 fill:#e8e8e8 style T2 fill:#90EE90 style T3 fill:#ffb3b3 style C1 fill:#87CEEB style C2 fill:#DDA0DD style C3 fill:#ADD8E6 style C5 fill:#90EE90 style C6 fill:#ff9999 style C7 fill:#ffb3b3 classDef noteBox fill:#fff9e6,stroke:#ffcc00,stroke-width:2px ``` # Troubleshooting Source: https://docs.cognite.com/cdf/infield/guides/troubleshoot Find information to help you troubleshoot common issues. ## I'm not seeing my checklist If you can't find the checklist you want to work with: Navigate to the Checklist tab. Select Ready, In Progress, or Done from the sidebar to sort the checklists by their status. If you can't find a checklist you've started working on, your supervisor may have set the checklist to Done. If you haven't started working on the checklist, contact your supervisor to verify that: * It has the right start and end time. * It's assigned to the right group or user. * It has the right status. Still not finding it? Contact your IT admin or Cognite Support. ## My device switches between the mobile and desktop versions If you want to set a default mode for your device: Navigate to Account > Manage account. Select Device mode and switch to the desired version of the application. If you use several devices, the setting won't affect another device's mode, and you can set the default version on the other device similarly. ## I can't enter negative values in my numerical readings If you want to add negative values or a special symbol to a numerical reading: Navigate to Account > Manage account. Select Device settings > Mobile and switch to the desired keyboard. Test the selected keyboard. ## I don't know what work order the checklist is created from If you want to see the details of a work order the checklist is created from: On the InField desktop, select Checklists. Open the checklist created from a work order. Select the checklist name displayed as a blue link. If the name isn't displayed as a blue link, the checklist was created from a template or its tasks. ## A checklist is missing from the schedule If you can't find a checklist in the [Schedules](/cdf/infield/guides/plan#view-scheduled-work) tab: Check that you are looking at the correct week. Ensure the template itself has a schedule defined. If the template isn't scheduled, it won't appear in the Schedules tab. Confirm the template has been moved from Draft to Ready. Only *Ready* templates generate scheduled checklists. ## I see two identical checklists for the same time This can happen if an old scheduling process is still running alongside the new system. Contact your supervisor or IT lead so they can turn off the **Old Scheduling Function**. # About Cognite InField Source: https://docs.cognite.com/cdf/infield/index Help field workers and supervisors efficiently plan and perform their tasks replacing paper-based data-gathering. As a **field worker**, use InField to complete your tasks and capture measurements and observations when you work. As a **supervisor**, use InField to plan, schedule, and assign work activities. ## Key workflows Create templates, add tasks and schedules to generate checklists automatically. Manage work orders and checklists, assign to teams or users, and track completion progress. Complete tasks, add measurement readings, and work offline on your mobile device. Record equipment malfunctions, track status, and send observations to SAP. ## Support and access Find solutions for common issues, such as missing checklists, device mode switching, etc. Access the InField application to start planning, scheduling, and completing your field work activities. # About data integrations Source: https://docs.cognite.com/cdf/integration/concepts/about_data_pipelines Learn about the three main components of data pipelines in CDF: data integration, data transformation, and data contextualization. To analyze and contextualize your data in Cognite Data Fusion (CDF), you need to establish efficient **data pipelines** between your existing data infrastructure and CDF. Data pipelines consist of three main components: data integration, [data transformation](#data-transformation), and [data contextualization](#data-contextualization). ## Data integration CDF offers two approaches for data integration: * **Extractors** are services that transfer data from external systems into CDF. Extractors perform one-way communication, moving data from source systems into CDF's specialized storage or the staging area (CDF RAW). * **Connectors** are services that enable bi-directional data transfer between external systems and CDF. Connectors can both read from and write to external systems, providing more flexibility in data synchronization. Depending on their implementation, extractors and connectors write data to different destinations in CDF. Some integrations write directly to specialized and optimized storage for specific resource types, such as time series, assets, and simulations. Other integrations write to [CDF RAW](/cdf/integration/guides/extraction/raw_explorer), the staging area for ingesting data. Some integrations support multiple destination options. Always refer to the specific [extractor](/cdf/integration/concepts/extraction) or [connector](/cdf/integration/guides/simulators/connectors) documentation to understand the available data destinations and capabilities for your integration. ## Data transformation The [data transformation](/cdf/integration/concepts/transformation) component shapes data to fit a CDF data model. When working with staged data in CDF RAW, you can use [Transformations](/cdf/integration/guides/transformation/transformations) to process and structure your data. The transformation component typically contains most of the data processing logic in your pipeline. ## Data contextualization The [data contextualization](/cdf/integration/concepts/contextualization) component establishes relationships between data elements from different source systems within your data model. It combines machine learning capabilities, a rules engine, and domain expertise to create meaningful connections between your data resources. ## Best practices Regardless of the tools you use, we recommend using a **modular** design for your data pipelines. This makes the pipelines easy to maintain and individual components easier to update or replace. If you're using extractors that write to CDF RAW, make sure that you lift the data out of the source systems and into the staging area before you start the data transformation. This reduces the load on your source systems. When you select an integration method, consider your specific needs. Extractors are ideal when you only need to move data from external systems into CDF, while connectors are better suited when you need bi-directional data synchronization between CDF and external systems. Write directly to specialized storage when your data is ready for immediate use, and use CDF RAW when your data requires additional transformation before reaching its final form. # About data integration with connectors Source: https://docs.cognite.com/cdf/integration/concepts/connections/about_data_integration_with_ connectors Learn how connectors enable bidirectional data integration between source systems and Cognite Data Fusion (CDF) with prebuilt Cognite connectors or custom solutions. The prebuilt connectors can **stream or sync data bidirectionally** between source systems and the [CDF staging area](/cdf/integration/guides/extraction/raw_explorer) or directly to a [CDF service](/dev/concepts/resource_types) with little or no data transformation. Connectors maintain data consistency between source systems and CDF through controlled `read` and `write` operations, with appropriate access permissions for bidirectional synchronization. ## Cognite connectors | Connector | Description | | - | - | | [PROSPER connector](/cdf/integration/guides/simulators/connectors/prosper/prosper_setup) | Integrates with the PETEX PROSPER simulator to create process twins for wells and pipelines in CDF. Run multiphase flow simulations with direct data exchange between PROSPER and CDF. | | [GAP connector](/cdf/integration/guides/simulators/connectors/gap/gap_setup) | Integrates with the PETEX GAP simulator to create process twins for production and injection networks in CDF. Run steady-state network modeling and optimization with direct data exchange between GAP and CDF. | | [PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway) | Ingests data into CDF from ETL tools. This service provides a PostgreSQL interface to CDF and the CDF resource types appear as tables in the PostgreSQL database. | | [SAP write-back](/cdf/integration/guides/interfaces/sap_writeback) | Creates or updates records in your SAP S/4HANA system. | ### Download installation files Navigate to the Data fusion workspace > Integrate > Extractors. Use the search bar to filter the list of data integration services and select the connector best suited to your integration needs. Download the **Installation file** and, optionally, the PDF documentation. Extractor download page ## Custom connectors You can create custom connectors if your integration requirements can't be met by the prebuilt connectors. ### Cognite Python extractor-utils Use the [Python extractor-utils](https://github.com/cognitedata/python-extractor-utils), an extension of the [Cognite Python SDK](/dev/sdks/python), to develop extractors in Python. For more information, see the [Python extractor-utils SDK documentation](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest). ### Cognite .NET extractor-utils Use the [.NET extractor-utils](https://github.com/cognitedata/dotnet-extractor-utils) to develop extractors in C#. For more information, see the [.NET utils documentation](https://cognitedata.github.io/dotnet-extractor-utils/tutorials/intro.html). ### Cognite .NET simulator-utils Use the [.NET simulator-utils](https://github.com/cognitedata/dotnet-simulator-utils) to develop simulator connectors in C#. For more information, see the [.NET utils documentation](https://cognitedata.github.io/dotnet-simulator-utils/docs/create-connector.html). # Cognite connectors overview Source: https://docs.cognite.com/cdf/integration/concepts/connections/get_started Explore prebuilt and custom connector solutions to integrate data between source systems and Cognite Data Fusion (CDF) with bidirectional synchronization capabilities. Stream or sync data bidirectionally between source systems and CDF with prebuilt connectors or build custom solutions for specialized integration requirements. Choose your integration path based on your data source and technical needs. Understand how connectors enable bidirectional data flow between source systems and CDF through controlled read and write operations. Integrate PETEX PROSPER simulator to create process twins for wells and pipelines with multiphase flow simulation capabilities. Connect PETEX GAP simulator for production and injection network modeling with steady-state optimization and analysis. Ingest data from ETL tools using PostgreSQL protocol with CDF resource types mapped as database tables. Create or update records in SAP S/4HANA systems directly from CDF with bidirectional synchronization. Develop custom extractors in Python or .NET using Cognite SDK utilities with built-in state management and error handling. # Data catalog Source: https://docs.cognite.com/cdf/integration/guides/exploration/data_catalog_concepts Search, explore, and manage data sets in a central and secure library of data assets. Use the **Data catalog** to get an overview of the data that's available to you, and use it as a starting point for exploring data resources. You can use the Data catalog as: * A part of your onboarding workflow, to get overviews of existing data, and to verify that newly onboarded data appears as expected. * An entry point for searching for and exploring data in CDF. * A way to manage data governance and user access to data resources. ## Before you start You must create data sets before the data is visible in the **Data catalog**. The data you see in the Data catalog also depends on your access rights, including organizational settings and which CDF projects you belong to. To view and manage data from the Data catalog, you need [capabilities](/cdf/access/guides/capabilities#data-sets-and-data-catalog) for data sets. Note that staged data (RAW), and some image and video data aren't included in the Data catalog. ## Get started Navigate to Data fusion > Data catalog. Search, explore, and manage data sets from the **Data catalog** page. ## Explore and manage data You can see an overview of your data sets and their associated data resources on the **Data catalog** page. You can create data sets, view details in data sets, and edit data sets on this page. Select Explore data from the **Data catalog** page to open the Data explorer. Here you can search, browse through, or manage data for resource types within a data set, like files, events, or time series. You can, for example: * **Preview** data to get an overview of your data. * **Navigate** to data and narrow your search using filters. * **View** data * Metadata on resources like descriptions, IDs, and governance status. * Where the data came from (data sources), when the data was created in CDF, and when it was updated. * Open and view files, like PDF images, and see metadata on those files. * **Manage** data * Create extraction pipelines, and see details about how data is ingested from the extractors. * Manage groups that have access to specific data sets. * Create new data sets. # Data explorer Source: https://docs.cognite.com/cdf/integration/guides/exploration/data_explorer Browse and search through resource types to find the data you need, with drill-down into details and connected resources. ## Get started Navigate to Data fusion > Data explorer. Search for name, description, content, metadata values, ID, external ID, event type and subtype, units, documents, etc. Select the Search configuration icon next to the search bar to fine-tune your search. The search results are sorted into tabs according to their resource types. When selecting a resource type, you can continue fine-tuning your results using the left-side panel. ## Explore the search results * Select an element to see details and connected resource types. * Select a time series to open a preview panel where you can zoom in and out and interact with the time series. * Select an engineering diagram to search for data in a specific diagram. * On each tab with a resource type, you can filter how many linked elements you want to see: * Select Directly linked + name of resource type to see data that are directly linked to the selected resource type. * Select Linked + name of resource type to see data that are directly linked to the resource type and all subelements of the directly linked data. * Select **Table** to add and rearrange columns in the search result. Add the 3D availability column on the Assets tab to inspect the equipment's surroundings in contextualized 3D models. You can also view the results in [Charts ](/cdf/explore/charts) and [Canvas ](/cdf/explore/canvas). ## Use the filters Use the Common filters on the left panel to filter the result across all the data, and the **resource-specific** filters below to add filters for the specific resource types. You can add several values to each filter. Common results are marked as **blue** in the table, and resource-specific results are marked as **green**. ### Filtering on metadata In the Metadata filter, you'll find several keys, each typically with several values. If you select several values for the same key, these are treated with the `OR` operation. For example: ``` Name=Pump123 OR Pump456 ``` Values for different keys are treated with the `AND` operation. For example: ``` Name=Pump123 OR Pump456 AND Description=Observation ``` Combining the Metadata filter with other common or resource-specific filters is treated with the `AND` operation. ## 3D functionality Use the 3D tab to see relevant 3D models. Select and drag to see the model in 360 view. Use the toolbar to slice, measure distance, emphasize selectable objects, and copy the URL. # Image and video data Source: https://docs.cognite.com/cdf/integration/guides/exploration/images_videos Upload and manage images and videos containing essential metadata and visual information for optimizing operations. **Deprecated**: You can continue to use the **Image and video management** interface during the deprecation period. After retirement, access your files through Data explorer, the Cognite API, or the Cognite SDK. See [Deprecated and retired features](/cdf/deprecated) for details and timelines. This can be the time or location of a photo, and information in the images, like a number on a tag or an image that shows rust on a pipe. You can upload and modify image and video data in CDF using the Image and video management user interface. When you work with image and video data in CDF, you can: * Upload images and videos. * Extract metadata embedded within the files and add new metadata. * Add and modify files and data in bulk. * Add or review annotations on images. * Analyze and explore data: * Apply filters and search for data. * Explore data in different views: list view, gallery view, or map view. * View images in full size. * Stream videos. The CDF user interface supports .jpeg, .jpg, .png, mp4, and .WebM files. If you need to upload other types of files, you can do that with APIs. ## Before you start To view and edit image and video data, you need [these capabilities](/cdf/access/guides/capabilities#data-explorer-and-image-and-video-data). ## Get started Navigate to Data fusion > Images and videos. The file overview loads up to 1000 images at a time. Use search filters to narrow the results to ensure you see and access the most relevant images. In the search result, select an element in the file overview to see image details and annotations. ### APIs for image and video data You can upload, enrich, and explore image and video data using the CDF user interface or the Cognite APIs. These are the primary APIs that CDF uses for this: * [Files API](/api-reference/concepts/20230101/files) - to store the data. * [Annotations API](/api-reference/concepts/20230101/annotations) - to annotate image data. # Explore image and video data Source: https://docs.cognite.com/cdf/integration/guides/exploration/images_videos_tasks Upload, enrich, and explore image and video data to gather insights for decision making and task optimization. **Deprecated**: You can continue to use the **Image and video management** interface during the deprecation period. After retirement, access your files through Data explorer, the Cognite API, or the Cognite SDK. See [Deprecated and retired features](/cdf/deprecated) for details and timelines. ## Upload image and video data Navigate to Data fusion > Images and videos. Select Upload to open the Upload new files to CDF dialog. Select Select directory to navigate to a set of files, or choose Select files to choose individual files. Or, drag files into the upload area of the window. Choose a data set from the drop-down list to upload data within a specific [data set](/cdf/data_governance/concepts/datasets). Select Upload files to upload the selected files. ### Import metadata during upload Exif, short for Exchangeable Image File, is the standard format for storing information in digital photography image files using JPEG compression. It includes metadata like: * Data and time the image was taken. * GPS information. * Information about the technical details related to the camera, like shutter speed, lenses, and exposure. You can import camera-generated metadata on the images you upload to CDF. The Extract Exif-data from files check box is selected by default. If you don't want to extract Exif metadata to CDF, deselect the check box. Not all images contain Exif data, for example, if the image file is a screenshot. ## Group and organize image and video data You can organize images and videos by creating custom metadata for a group and classify information. For example, you can create a directory with image files for a specific location. And you can organize data by adding metadata for further classification. For example, you can add a metadata key. like *category*, and then give that key values, like *equipment* or *measurements*. Grouping and organizing image and video data improves search and can be used to prepare the data for machine learning to build custom applications for automating operations. There are different types of metadata associated with images and videos in CDF. Metadata can be: * Data imported with the image at upload, like Exif data. * Information created during the upload process, like the date and time that the image was uploaded. * Custom-defined metadata created in CDF, like a new directory for image files. * Annotations created in CDF after the image is uploaded. You can add and edit metadata in the File Details dialog in the Image and video management window. You can also link the file to assets from File Details. Organizing data into groups and adding metadata labels to those groups makes it possible to apply search filters in the Filter dialog in the Image and video management window. The better organized your data is, the better your search results will be and the more value you get from the data in the system. ## Add annotations to images You can enrich images using annotations by marking specific regions of interest on an image. For example, you can add an annotation called *corrosion* and use that annotation to mark images that show rust on pipes or other types of corrosion. Any CDF user with the necessary permissions can use the same annotation to indicate regions on an image that show corrosion. When you filter your search to include these annotations, you can find and review all images with this annotation label. ### Annotation types There are two main types of annotation: **shapes** and **keypoint collections**. | Annotation type | Description | | - | - | | Shapes | Use the *bounding box* or the *add polygon* tool to select an area in an image and draw a shape around it. | | Keypoint collections | Use the *add point* tool to create a grouping of individual points on an image, for example, points that span the start and end of a needle on a gauge in a photo. | ### Create consistent annotation labels We recommend that you establish an organized approach and consistent naming convention to creating annotation labels and keypoint collection names for your images in CDF. Suppose you have several annotation labels for the same type of annotation, for example *ID*, *identification*, and *tag number*. In that case, search results will be less relevant than if all the annotations have one common name, like *ID*. CDF lets you choose from a drop-down list of pre-defined annotation labels and collection names when you create annotations. If you don't see the pre-defined label you need, you can create a new one. After you create it, your new annotation label appears in the pre-defined list. Make sure you have a structured approach to annotations before you create them. After you select Done, you can't delete your new annotation from the pre-defined list. ### Create annotation Select an image from the Image and video management window. Open the Review file window by selecting Review annotations or View file, depending on your view (List view or Gallery view, for example). Choose an annotation tool from the toolbar. You can, for example, draw a box around a specific region on an image that has surface anomalies, like *corrosion*. Or, you can add a collection of points to an image to indicate a keypoint collection. When you start using the annotation drawing tools, the Create annotations button appears. For consistency and improved search, use the pre-defined labels if they exist. After you create an annotation, you can see the annotation label in the annotations list for an image on the File Details tab. When you go to the Image and video management overview window, you see that the image now includes the annotation. You can use the Filter dialog to search for the image and other images with the same annotation label. ## Search image and video data You can narrow search results to find relevant images and videos by applying filters in the Filter result dialog in the Image and video management window. You can specify to search for: * Date and time. * Asset. * Annotation and annotation state. * Media type (image or video). * Data set. * Directory prefix. ## View images in full scale To view an image in full scale from the List view in the file overview, select the image you want to view, and then select Review annotations. To view an image in full scale from the Gallery view in the file overview, hover over the image and select View File. You can scale up and down inside the Review window to increase or decrease image viewing size. You can change or add new metadata for images in this window. ## Review annotations You can approve and reject annotations from **CDF** > **Image and video management** > Review for a selected image. You see a list of annotations for a selected image on the Annotations tab. To approve an annotation for an image, go to the Annotations tab for the image, open the annotation, and select the **True** button. To reject an annotation, select the False button. Rejecting an annotation doesn't delete the annotation. Approving and rejecting annotations enables collaboration on image analytics. You can add annotations that are reviewed later by a domain expert. Approving and rejecting annotations can also be used for searching and filtering data, say, to find all images with the label *corrosion* that have been approved. ## Modify image data You can view and edit details about files in the File Details dialog for a selected image. Here you can: * Add and edit the external ID. * Attach labels to the file. * Link to the file to assets. * Add or modify metadata associated with the file. ### Modify data in bulk Select several images from the Image and video management window, and select the Bulk edit drop-down, where you can choose to edit, review, download, or delete data in bulk. The Bulk edit files dialog lets you select data types and actions to edit. You can: * Edit shared metadata on the selected images, for example, a title of a document. * Add labels from your existing set of labels, for example, the name of a project. * Add new assets, or remove existing asset links. * Update the source of the files. For example, update the name of the application where the files originated. * Update the name of the directory that the files belong to. * Update the status of annotations for the selected files. For example, set them to *approved* or *rejected*. * Delete annotations for the selected set of files. ## Stream videos You can upload videos using the Upload button in the Image and video management window and stream videos directly in CDF. Standard video playback functionality is available for your video data in CDF, including play, pause, change volume, and view full-screen. ## Troubleshooting ### Issue: File doesn't upload **Possible resolutions** * **File format**: Check that you are uploading supported file formats. Supported file formats are: .jpg, .jpeg, .png, .mp4, and WebM using the user interface. If you need to upload a different type of image, you can do that using an API. * **File size**: The maximum amount of files to upload via the user interface is 250, and a total of 1GB. Check to see if you are attempting to upload more data than the user interface supports. You can also upload data using an API instead of the user interface. # Setup and administration for Microsoft Entra ID Source: https://docs.cognite.com/cdf/integration/guides/extraction/admin_oidc Register and configure apps in Microsoft Entra ID for use with data integration services such as extractors and connectors. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to Cognite Data Fusion (CDF) data securely. To perform the steps below, you must be an administrator of Microsoft Entra ID. ## Before you start Make sure you have [registered the Cognite API and the CDF in Microsoft Entra ID](/cdf/access/entra/guides/configure_cdf_azure_oidc) and [set up Microsoft Entra ID and CDF groups](/cdf/access/entra/guides/create_groups_oidc) to control access to CDF data. ## Step 1: Register an app in Microsoft Entra ID Cognite recommends that you register **one app per integration service and per environment**. For example, we recommend 3 app registrations to run one extractor or connector in *dev*, *test*, and *prod* environments. Sign in to the [Azure portal](https://portal.azure.com) as an admin. If you have access to multiple tenants, use the Directory + subscription filter in the top menu to select the tenant in which you want to register an application. Search for and select **Microsoft Entra ID**. Under **Manage**, select **App registrations** > **New registrations**. In the **Register an application** window, enter the app name and select **Register**. Copy and make a note of the **Application (client) ID**. You'll need this value for authentication when reading and writing data with your integration service. Under **Manage**, select **Certificates & secrets** > **New client secret**. OIDC client secret configuration Enter a client secret description and an expiry time, and then select **Add**. Copy and make a note of the client secret in the **Value** field. Make sure you copy this value now. This value will be hidden after you leave this page. ## Step 2: Create a group in Microsoft Entra ID and add the registered app as its member Open the overview window in Microsoft Entra ID and select **Manage** > **Groups**. Create a group. Read more [here](/cdf/access/entra/guides/create_groups_oidc). Open the group. Under **Manage**, select **Members** > **Add members**. Find the app you created above and click **Select**. Add all **users** you want to have access to the integration service as members. Return to the overview, and then copy and make a note of the **Object Id**. OIDC app group configuration showing Object ID ## Step 3: Create a group in CDF and link to the Microsoft Entra ID group Sign in to CDF as an admin and navigate to **Access** > **Groups** > **Create new group**. In the **Create a new group** window, enter the **group name** (case sensitive). Add the necessary capabilities for your integration service: * For extractors, see [extractor capabilities](/cdf/access/guides/capabilities#on-premises-extractors). * For connectors, see [connector capabilities](/cdf/access/guides/capabilities#simulator-connectors). Link the group to a Microsoft Entra ID group: * In the **Source ID** field, enter the **Object Id** for the AAD group exactly as it exists in AAD. You can use the same group Id for multiple services. * In the **Source name** field, enter the name of the group in Microsoft Entra ID. ## Step 4: Deploy your integration service Configure and deploy your integration service according to the documentation: * For extractors, see the [extractors documentation](/cdf/integration/concepts/extraction#download-installation-files). * For connectors, see the [connectors documentation](/cdf/integration/guides/simulators/connectors). # Set up on-premises data integration Source: https://docs.cognite.com/cdf/integration/guides/extraction/security Best practices for setting up secure on-premises data integration services for Cognite Data Fusion (CDF), including user configuration, secrets management, and network security. In CDF, there are two options for integrating with external systems: extractors for one-way data transfer into CDF, and connectors for two-way communication between external systems and CDF. ## User and service configuration Most data integration services deployed on Windows run as Windows services, which provides reliable and stable operation. Each deployment should ideally have a dedicated Windows user with minimal necessary permissions. The service user must be granted: * `Read` access to the service executable * `Write` access to the working directory ### Configure access in Windows Right-click the file or folder you want to configure and select **Properties**. Select **Security**. Select **Edit** and add the service user to the groups. Select the appropriate permissions for the user. For Linux systems, see `$ man chmod` for instructions on how to change permissions to specific files and folders. ## Configuration files and secrets Configuration files should generally be kept free of secrets. Use environment variable substitution or Azure Key Vault instead. Note that this is *not* itself more secure, but it makes it harder to accidentally leak secrets when sharing configuration files. Environment variables should be assigned to the service user. Do *not* store secrets in system environment variables. ### Using Azure Key Vault If you're using Azure Key Vault, the client credentials for the user should only have access to the secrets necessary for that service. In general, using Azure Key Vault is *less* secure than storing secrets on-premises. Anyone with access to the on-premises machine will also have access to the Azure Key Vault secrets, but now the secrets are also accessible from other locations. The primary advantage of Azure Key Vault is that it allows you to rotate secrets easily. ## Network configuration ### Firewall configuration Cognite's data integration services don't require any open ports, unless noted otherwise in the setup documentation. If you do open ports, for example, for Prometheus metric scraping, make sure that only the specified service is accessible on that port. Set `Program` in the firewall rule to the service executable. If the external application accessing that port has a static IP address, you can also set the `Remote address` option in the firewall rule. ### Transport Level Security The data integration services use Transport Layer Security (TLS) when connecting to CDF. Since traffic to CDF always goes over the internet, it's very important to have proper end-to-end encryption. This is required and can't be disabled. When connecting to CDF, the service verifies that a valid certificate authority issued the CDF certificate. This is to ensure you're connecting to CDF and not to something pretending to be CDF. Certificate validation may fail for various reasons, each with separate solutions: * **Proxy or firewall with invalid certificate**: There's a proxy or firewall in between systems with an unsigned or invalid certificate. Add the certificate for the proxy to the certificate store. In Windows, open the certificate and select **Install Certificate**. * **Machine lacks up-to-date certificates**: The machine lacks up-to-date certificates and is likely far behind on Windows updates. **This is a severe security vulnerability**. *Always* keep your Windows updates current. ### Source system security Security varies between different integration services when you're connecting CDF to source systems. Some systems use TLS, in which case the recommendations above apply. Other services have more specialized security features for the specific source. Read the service documentation to determine if any action is necessary to ensure a secure connection. # About the PostgreSQL gateway Source: https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway/postgres_gateway_basics Learn about the PostgreSQL gateway and how to ingest data into Cognite Data Fusion (CDF) from ETL tools. The Cognite PostgreSQL gateway provides a PostgreSQL interface to CDF, and CDF resource types appear as tables in the PostgreSQL database. You can ingest data directly into CDF resources, like assets, events, and datapoints, and to [CDF RAW](/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw). ETL data flow diagram showing data movement from source to CDF * Microsoft's Azure Data Factory is the officially supported ETL tool. Alternative tools may be used, but without support from Cognite. * The PostgreSQL gateway is primarily intended for ingestion. You can run a simple filtered read in a SQL client to verify ingested data. Aggregations, joins, and `ORDER BY` on fields the CDF API doesn't support sorting on aren't pushed down and can trigger a full table scan. See [Limitations](#limitations). * Don't connect business intelligence tools to the gateway. For BI or aggregation-heavy queries, use [Transformations](/cdf/integration/concepts/transformation) to materialize data to CDF RAW or a data model, then query that result. Don't query the materialized data through the PostgreSQL gateway. ## When to use the PostgreSQL gateway Consider using the PostgreSQL gateway if: * You're integrating a new data source that can be accessed through Azure Data Factory (ADF) or another ETL tool that supports writing to PostgreSQL. You can use the PostgreSQL gateway as a sink in ADF. * You have previously built an extractor that can push data to PostgreSQL, but not to CDF. Consider other solutions if: * You need aggregations, joins, or a business intelligence tool. Use [Transformations](/cdf/integration/concepts/transformation) to materialize data, then query that result. Don't query the materialized data through the PostgreSQL gateway. * You need very high performance, especially for ingestion of RAW rows or time series datapoints, in the order of tens of thousands rows/points per second (10-50k/s as a ballpark figure). To verify ingested data, use a SQL client such as [`psql`](https://www.postgresql.org/docs/current/app-psql.html). Connect to the PostgreSQL gateway using the credentials returned from the [Postgres Gateway Users](/api-reference/concepts/20230101/postgres-gateway-users) endpoint: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} psql -h [host] -U [user] -d [user] ``` The name of the database you connect to is identical to the username. When prompted for a password, use the password returned by the create users API. ## How the PostgreSQL gateway works The PostgreSQL gateway is a [foreign data wrapper](https://www.postgresql.org/docs/current/ddl-foreign-data.html), which is a PostgreSQL extension where you create **foreign tables**. These are constructs that look like normal PostgreSQL tables, but instead are thin references to external data. When you query a foreign table in the PostgreSQL gateway, your query is translated into one or more HTTP requests to the Cognite API. This implicitly creates limitations. The PostgreSQL gateway is only capable of doing what the Cognite API can do, but will typically try to fulfill any query. If you write a query that doesn't easily translate into requests to the Cognite API, the gateway may read *all* available data from CDF to fulfill your request. For example, this query reads *all* assets from CDF, then finds the asset with the highest ID: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT * FROM assets ORDER BY id DESC LIMIT 1 ``` The Cognite API doesn't support [ordering by ID](/api-reference/concepts/20230101/assets). If you order by `name`, the gateway translates the query into a single request to CDF: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT * FROM assets ORDER BY name DESC LIMIT 1 ``` ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sort": { "property": ["name"], "order": "desc" }, "limit": 1 } ``` A simple query can still result in a slow and expensive series of requests to CDF. Use `EXPLAIN` to see what the query will do: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EXPLAIN SELECT * FROM assets ORDER BY name DESC LIMIT 1 ``` The query plan contains: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Summary: Execute 1 CDF query Operation: Query with no filter Limit: 1 Sort: name descending, nulls first, collation: default ``` This plan means the gateway queries CDF with a sort clause on `name`. If `EXPLAIN` shows many CDF requests, or that the gateway will read all rows to fulfill the query, don't deploy that query. ### Custom tables In the Cognite API, you can create custom tables that represent [data modeling views](/cdf/dm/dm_concepts/dm_containers_views_datamodels) or [RAW tables](/cdf/integration/guides/extraction/raw_explorer). This is a convenient interface for ingesting data into types that have a flexible schema in CDF. When you create a custom table representing a RAW table, the table in CDF RAW isn't required to pre-exist. Instead, the table is automatically created if you *insert* data into the table in the gateway. If you delete a table in the gateway, the corresponding resource isn't deleted. When you create a custom table representing a data modeling view, the view is required to exist when the table is created since the data modeling definition is imported. If the view is deleted, the table in the gateway isn't deleted. ## Limitations The PostgreSQL gateway only does what the CDF API can do. Simple filters push down. Aggregations, joins, and some `ORDER BY` clauses don't, and can read all matching data from CDF. Operations such as `UPDATE ... WHERE` and `DELETE ... WHERE` work as expected. Filters are pushed down as much as possible. This query performs a single `update` request: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} UPDATE assets SET name = 'new name' WHERE external_id = 'test' ``` This query first finds assets with a filter on `name`, then updates based on the result: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} UPDATE assets SET name = 'new name' WHERE name = 'old name' ``` ### Errors from CDF Note that in general the PostgreSQL gateway does *not* automatically ignore errors from CDF. If you try to update assets that don't exist, the query will fail. This also applies for attempting to insert duplicates. These are *not* translated on a per-row level to PostgreSQL rejections, and queries like `UPDATE ... ON CONFLICT DO ...` won't work. ### Joins and aggregations Joins and aggregations aren't pushed down to the CDF API. This query retrieves *all* assets from CDF: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT COUNT(id) FROM assets ``` The PostgreSQL gateway doesn't push down joins at all. Large joins can be slow and expensive. ## Best practices These are the best practices to apply when using the PostgreSQL gateway. * Use `COPY FROM ...` for insertion as this is more efficient. If this isn't possible, use batch `INSERT` queries. Avoid making many `INSERT` queries with one row in each. * Run `EXPLAIN` on queries before you deploy them. Avoid queries that result in full-table scans or many CDF requests, unless that's what you want. * Ingest directly into CDF resources when you can. This simplifies the ingestion pipeline and avoids using Transformations only as an extra ingestion step. ## Further reading * [Setup and administration for PostgreSQL gateway 2.0](/cdf/integration/guides/interfaces/postgres_gateway) * [Use Azure Data Factory with PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway/setup_data_factory) * [Ingest data into CDF RAW](/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw) * [About transformations](/cdf/integration/concepts/transformation) * [Postgres Gateway Users](/api-reference/concepts/20230101/postgres-gateway-users) # Data sampling and validation for time series input Source: https://docs.cognite.com/cdf/integration/guides/simulators/about_data_sampling Learn how data sampling and validation ensure simulation runs with reliable input data through time concepts, validation methods, and best practices. ## Time concepts The system uses these time-related concepts to manage the data sampling and validation process: * `run time` is the reference point for all data preprocessing activities. This is user-defined and defaults to when the simulation is triggered or when the schedule executes. * `validation window` is a user-defined duration that's only relevant when data validation is enabled. The window represents the time range before the runtime where the system searches for valid data. * `validation end` is equal to the runtime. * `validation start` is calculated as `validation end` minus `validation window`. * `sampling window` is the user-defined duration that determines the time range for sampling process data. This is only used when data sampling is enabled. * `sampling end` equals the runtime if data validation is disabled. Otherwise, it's determined based on valid time ranges after data validation. * `sampling start` is calculated as `sampling end` minus `sampling window`. * `simulation time` represents the middle point of `sampling window` if data sampling is used. Otherwise, this equals the runtime. This timestamp is associated with your simulation results. ## Data sampling There are three approaches to data sampling, each with benefits and trade-offs: **Disabled data sampling**: No data validation is performed. The simulation time is equal to `run time`, and the latest datapoints before `run time` are used for all input time series. This is the simplest option, but it might miss data quality issues. Diagram showing data sampling disabled with latest datapoints before run time **Data sampling without validation**: The system uses the sampling window to sample data between `run time` and `sampling start`, taking the average value inside the window. `simulation time` is located in the middle of `sampling window`. This option provides data smoothing but no quality checks. Diagram showing data sampling without validation using sampling window for average values **Data sampling with validation**: The system searches for valid time ranges inside `validation window` to determine if there's a continuous time range up to the size of `sampling window` where process data can be sampled. The most recent valid time range is used as the sampling window, where data from the input time series is sampled as before. This is the most robust option for ensuring data quality. Diagram showing data sampling with validation using valid time ranges within validation window ## Data validation methods Use the data validation methods **Logic validation** and **Steady State detection** to check for valid process conditions before you select data for simulator runs. These methods can be used independently or combined to ensure appropriate simulation conditions. ### Logic validation Logic validation checks if a specific condition is met in the time series data. Use this to verify that a pressure reading stays above a certain threshold, or that a given equipment is online. You specify the time series, logical operator, and value to check against. The system marks time periods as valid with a 1 when your condition is met, and invalid with a 0 when it isn't met. The image shows an example of the logic validation applied to some sensor data. The **validation window** is 6 hours, the **logical operator** is greater, and the **value** is 45.0. The yellow-hatched areas identify the time regions where the logical condition was met. Example of logic validation with yellow-hatched areas showing time regions where conditions were met ### Steady-state detection Steady-state detection looks for periods where your process is in a pseudo-steady state. This is important when sampling data for steady-state simulations. You specify the time series and the parameters: `min section size`, `var threshold`, and `slope threshold`. The system marks time periods which were identified as “steady” with **1**, and “transient” periods with **0**. The detection method first splits the time series data points into statistically homogeneous segments using the *ED Pelt* change point detection algorithm. Each segment is then analyzed to determine if it represents a steady or transient period based on these parameters: * `min section size` specifies the minimum number of consecutive data points required for a segment to be considered in the analysis. This ensures that very short fluctuations aren't mistakenly identified as steady states. * `var threshold` sets the maximum allowed normalized variance within a segment. If a segment's normalized variance exceeds this threshold, it is classified as transient (0). This parameter helps identify regions where data remains consistently stable. * `slope threshold` defines the maximum allowed rate of change within a segment. The algorithm fits a line to each segment and compares its slope to 10 raised to the power of this threshold. Segments with slopes exceeding this value are marked as transient (0), helping to distinguish between truly steady regions and gradual changes. This image shows an example of steady-state detection applied to some sensor data. The **validation window** is 6 hours, the **min section size** is 15, the **var threshold** is 3.5, and the **slope threshold** is -3.0. The yellow-hatched areas identify the time regions where the steady-state criteria were met. Example of steady-state detection with yellow-hatched areas showing time regions meeting steady-state criteria ## Combined validation process Sampling time is calculated with the results of the data validation methods: * First, the valid timestamps are identified by calculating the intersection between all enabled data validation routines (when both return 1). * Then, the system checks if a continuous time range greater or equal to the sampling window size exists within the valid timestamps. The most recent time range with a size equal to the sampling window is selected to sample the input variables required for the simulation. This image shows an example of the data validation process and the identification of the sampling time. The logical check and steady-state detection methods are enabled and identify the valid timestamps, marked as blue-hatched areas in the third plot. The timestamps are used to find the sampling time, marked as a green-hatched area in the fourth plot. Data validation process showing valid timestamps and sampling time identification with blue and green-hatched areas ## Best practices for data sampling and validation * Consider the process characteristics when you set thresholds: * Align sampling windows with your process dynamics * Use steady-state detection for simulations requiring steady conditions * Apply logical checks when specific operating conditions must be met * Balance between data quality and simulation frequency: * Stricter validation leads to more reliable results but potentially fewer successful runs * Consider sensor reliability and data quality when setting validation parameters * Monitor validation success rates to optimize settings Proper data validation ensures simulation runs with reliable, stable input data, leading to more accurate results. However, it's important to find the right balance between data quality requirements and operational needs for your specific use case. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_configuration Learn how to configure the GAP connector with YAML parameters for automation and OpenServer connection settings. The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. To configure the GAP connector, you must create a configuration file with the parameters listed below and the [common connector configuration parameters](/cdf/integration/guides/simulators/connectors). The file must be in [YAML](https://yaml.org) format. ### Minimal YAML configuration file Use the minimal configuration file as a starting point for your configuration settings. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: ${COGNITE_PROJECT # The API host to connect to example: https://api.cognitedata.com host: ${COGNITE_HOST} # Configure the number of retry attempts for network requests to CDF. # The following configuration results in approximately 5 minutes of exponential backoff # before ultimately failing. cdf-retries: max-retries: 12 max-delay: 60000 # This is for Microsoft as IdP, to use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} connector: # Enter the data set ID to read the model files and simulator routines from. # Result inputs and outputs will be written to this data set. (required) data-set-id: ${COGNITE_GAP_DATASET} ``` ## Connector Several connector settings can be configured in the `connector` section of the configuration file. The only required parameter is `connector.data-set-id`. | Parameter | Description | | - | - | | **connector.data-set-id** | Enter the data set ID to read the simulation resources from. Result inputs and outputs will be written to this data set. | ## Automation This section includes the parameters for configuring the PETEX OpenServer connection. This is configured in the `automation` section of the configuration file. | Parameter | Description | | - | - | | **automation.terminate-process** | Set to terminate the GAP and OpenServer process at the end of usage (model revision validation / simulation run). | | **automation.license-hold-time** | Time in seconds to hold the license for after an action (model revision validation / simulation run). | | **automation.simulator-retries** | Set the simulator connection retries. | | **automation.simulator-retries.max-tries** | Maximum amount of times to retry a connection failure to the simulator. | | **automation.simulator-retries.max-delay** | Maximum delay before each retry attempt to the simulator. For example, "10s" , "1m". | | **automation.open-server-async-command-timeout** | Maximum number of seconds to wait before timing out a "DoCommand" operation. For example, "600s". | | **automation.simulator-executable-path** | Path to the GAP executable. When set, the connector manages the simulator process lifecycle directly. Use official PETEX binaries because the connector verifies the executable's digital signature before launching it. | | **automation.skip-undefined-numeric-outputs** | If `true`, numeric outputs that GAP returns as `NaN`, `Infinity`, or `-Infinity` are logged and omitted from the simulation results. The simulation run continues. If any value in an array output is `NaN`, `Infinity`, or `-Infinity`, the connector omits the whole array. Default value is `false`. | ## Recommended configuration for simulation stability For stable runs, let the connector manage the simulator process lifecycle. When you set `simulator-executable-path`, the connector starts and terminates the simulator processes, which reduces orphaned processes and licenses being held after runs complete. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} connector: automation: simulator-executable-path: "C:\\Program Files\\Petroleum Experts\\IPM 12\\gap.exe" license-hold-time: 120 terminate-process: true ``` By default, the connector only obtains a license when needed. However, if the license hold time is set to a period more than 60s, the connector retains the license for the specified period after the last simulation run or model revision validation. This improves throughput for batched simulation runs by avoiding the overhead of repeatedly opening and closing the simulator. # Parse GAP model revisions Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_model_parsing Learn how the GAP connector extracts information models from simulator model revisions including flowsheet hierarchy, equipment properties, and connections. The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. The GAP connector extracts the information model from GAP simulator model revisions. This feature parses the **flowsheet hierarchy** of GAP simulator model revisions, including **functional blocks**, such as process equipment, **block properties**, such as operating parameters, physical properties and configuration settings, **connections** between blocks, and **graphical information** for visualization processes. The connector stores the parsed information in Cognite Data Fusion (CDF) alongside the associated simulator model revision. ## Enable model extraction By default, model extraction is disabled. Enable the feature by adding the following to your `config.yml`: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} automation: information-model-extraction-enabled: true ``` ## Prerequisites Model parsing automatically activates when: * A valid `modelparsing.config.yml` file exists in the connector's configuration directory. * Connector configuration `automation.information-model-extraction-enabled` is set to `true`. * A new simulator model revision is created or an existing model revision is reparsed. * The GAP model contains a **single** flowsheet. ## Configure parsing You configure the model parsing in a separate YAML file (`modelparsing.config.yml`) that defines which properties to extract from different equipment types. Use `modelparsing.config.example.yml` for guidance. ### Configuration file structure ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Common properties extracted from all equipment types common-properties: - name: OilRate value-type: DOUBLE unit: quantity: Liquid Rate description: Oil rate from the network solver. address-suffix: .SolverResults[0].OilRate - name: WaterRate value-type: DOUBLE unit: quantity: Liquid Rate description: Water rate from the network solver. address-suffix: .SolverResults[0].WaterRate # Equipment-specific properties equipment-properties: - equipment: well properties: - name: WellModel value-type: STRING description: Type of well model used. address-suffix: .WellModel - name: TypeWell value-type: STRING description: Type of well. address-suffix: .TypeWell - equipment: pipe properties: - name: PipeModel value-type: STRING description: Type of pipeline model used. address-suffix: .PipeModel - name: PipeCorr value-type: STRING description: Type of pipeline correlation used. address-suffix: .PIPECORR ``` ### Property configuration parameters | Parameter | Description | Required | | - | - | - | | **name** | The display name for the property. | Yes | | **value-type** | The data type of the property. Supported values: `STRING`, `DOUBLE`, `STRING_ARRAY`, `DOUBLE_ARRAY`. | Yes | | **address-suffix** | The GAP address suffix to append to the equipment address to retrieve the property value. | Yes | | **unit.quantity** | The unit quantity associated with the property. Only relevant for numeric properties. | No | | **description** | A brief description of the property for documentation purposes. | No | ## Extract information This is how the GAP connector extracts information from simulator model revisions: When a simulator model revision is validated, the connector loads the GAP model. The parser queries GAP to discover all equipment types and instances in the model. For each equipment instance, the parser: * Extracts common properties defined in the configuration. * Extracts equipment-specific properties based on the equipment type. * Retrieves unit information for numeric properties. The parser analyzes equipment connections to generate edges representing material flows The complete information model is stored in CDF and associated with the simulator model revision. ## Access parsed information When a simulator model revision has been parsed, use the simulator model revision endpoints to fetch the associated flowsheet data. ## Property extraction errors During model parsing, the connector automatically skips any property where it fails to extract data to continue parsing. For example: **Invalid address** If the `address-suffix` contains a typo, for instance, `.SolverResults[0].OliuRate` instead of `.SolverResults[0].OilRate`, the property is skipped. **Incorrect unit quantity** If the specified `unit.quantity` doesn't match the property type, for instance, using `Temperature` for a pressure value, the property extraction fails and is skipped. **Wrong value type** If the `value-type` doesn't match the actual data type, for instance, specifying `DOUBLE` for a string property, the property is skipped and the connector logs details about the failed property extraction, continues processing remaining properties, and excludes the failed property from the final information model. ## Limitations and considerations * Only GAP models with a **single flowsheet** are supported. The connector skips parsing for models with multiple flowsheets. * Model parsing requires a **valid configuration file**. * Model parsing **adds processing time** during model validation. The impact depends on model complexity. * Not all properties may be available for all equipment types. Missing properties are logged and skipped. * **Unit information** is retrieved when available, but may be missing for some properties. * Individual property extraction failures don't stop the overall parsing process, but may result in incomplete information models. # System requirements Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_server_requirements Review hardware, software, and network requirements for installing the GAP simulator connector on physical or virtual machines. The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. You can install the connector on a physical server, on a virtual machine on-premises, or in the cloud. ## Memory The default configuration requires the server to have a minimum of **8 GB RAM** and an additional **256 GB HD** for logging and storage. You can adjust the configuration to reduce memory usage. ## CPU For the default values in the configuration, the server needs a minimum of **4-core CPUs**. The memory and CPU requirements also depend on the requirements provided by GAP. The machine should fulfill the basic requirements needed to run GAP simulators. ## Software * Windows Server 2016 or later. * A Windows account with admin rights to install the connector. * Access to the Windows machine, for instance through VPN. * .NET 8.0 runtime. * PETEX OpenServer license. * The PETEX simulators; GAP or GAP. The connector has been tested on PETEX IPM v11.5 - v13.5. ## Dedicated environment The GAP connector requires a **dedicated environment** where it's the only service interacting with OpenServer. No other processes, connectors, third-party applications, or custom integrations can use OpenServer in the same environment. If you encounter issues while running the connector in an environment with other OpenServer processes, we may be unable to offer support. **Recommendation:** Install the GAP connector on a **dedicated virtual machine (VM)** or **physical server**. ## Network access * If the VM is behind a firewall, allow access to the endpoint: `https://{cluster}.cognitedata.com`. For example, if the project is in the Europe 2 (Microsoft) cluster, then this endpoint is [https://westeurope-1.cognitedata.com](https://westeurope-1.cognitedata.com). # Setup Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_setup Learn how to set up and configure the GAP connector Windows service, including connection modes and model management. The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. ## Before you start * Check the [server requirements](/cdf/integration/guides/simulators/connectors/gap/gap_server_requirements) for the connector. * Make sure the connector has [access capabilities](/cdf/access/guides/capabilities#simulator-connectors) in a Cognite Data Fusion (CDF) project. * Create a configuration file according to the [configuration settings](/cdf/integration/guides/simulators/connectors/gap/gap_configuration) in the working directory of the folder where the connector is installed. The file must be in YAML format. ## Connect to the GAP simulator The GAP connector connects to the PETEX OpenServer protocol via COM, which connects with the GAP simulator. For any connection errors, you can define retry parameters in the configuration file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} automation: simulator-retries: max-tries: 12 max-delay: "60s" ``` This configuration enables retry attempts to the simulator for up to 5 minutes before failing. The retries follow an exponential backoff strategy, starting with shorter delays and gradually increasing to a maximum interval of one minute. | Retry attempt | Delay duration before attempt | | - | - | | 1 | ░░ (0.5s) | | 2 | ░░░░ (1s) | | 3 | ▓▓▓▓▓ (2s) | | 4 | ██████ (4s) | | 5 | ██████████ (8s) | | 6 | ██████████████ (16s) | | 7 | ██████████████████ (32s) | | 8 | ████████████████████████ (60s limit) | | 9 | ████████████████████████ (60s limit) | | 10 | ████████████████████████ (60s limit) | ## GAP connector modes The GAP connector supports two modes: * **Single bundle mode (.gar files)**: A simple approach with all components packaged in a single archive file. * **External dependencies mode (.zip files)**: For advanced users to manage network models and dependencies separately for greater flexibility and control. Manage simulation models via a single bundle mode using `.gar` files. These files are PETEX's standard for a GAP archive file. A `.gar` file is a self-contained archive that includes the GAP surface network model along with all associated IPM files, such as well models, tanks, and other components required for simulation. #### Prerequisites * The `.gar` file must contain a valid production network model as the primary entry point. * All network models (production and injection) referenced in the system must be present. * All dependent files (wells, pipes, tanks) referenced by any network must be included in the `.gar` file. * Missing dependencies will invalidate the model and prevent simulation execution. #### Connector workflow The connector downloads and extracts the contents of the `.gar` file into a temporary directory. The connector searches for a production network model, which is indexed as `MOD[0]` or identified by the `{PROD}` literal constant. This serves as the primary entry point. This network model is mandatory and must be present in the `.gar` file. From the production network, the connector identifies any dependent networks such as Water Injection (WINJ), Gas Injection (GINJ), or GasLift Injection (GLINJ) networks that are referenced in the model. For each identified network (production and injection): * The connector locates all dependent files for wells, pipes, and tanks referenced by that network. * All dependent files must be present within the `.gar` archive. * If any required dependent file is missing, the model is considered invalid and can't be used for simulations. When all dependencies are validated, the connector updates file paths for all components to reference their new locations within the extracted directory. Manage simulation models via an external dependencies mode (.zip files). Use this mode when you need to **frequently update individual components**, such as individual well models, VLP tables, or other components without repackaging the entire model. This mode **reduces storage** since it avoids duplicating unchanged files across model revisions and enables different team members to work on different components independently. #### Prerequisites * Upload a `.zip` file that only contains GAP network files (production and injection networks). * Map individual node dependencies (`.OUT`, `.VLP`, `.MBAL` files) as separate CDF files * Map external dependencies with the correct file IDs and argument paths. * Make sure all referenced external dependency files exist in CDF. * Specify the GAP model path for each dependency in the `arguments` field. #### Connector workflow The connector downloads and extracts the `.zip` file containing only the GAP network files (production and injection networks). The connector reads the external dependencies list provided with the model revision, which maps node dependencies to individual CDF files. Based on the mapping, the connector downloads each external dependency file (`.OUT`, `.VLP`, `.MBAL`) from CDF separately. The connector uses the `arguments` field in the external dependencies to resolve file paths. For example: * `"GAP.MOD[{PROD}].WELL[{A1}].File"` maps to the PROSPER file for well A1 * `"GAP.MOD[{PROD}].WELL[{A1}].VLPFile"` maps to the VLP file for well A1 The connector combines the GAP network files with all external dependencies to create a complete simulation model ready for execution. #### Example: External dependencies structure When you create a model revision with external dependencies, provide a list of dependencies with this structure: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalDependencies": [ { "file": {"id": 1231231231}, "arguments": { "address": "GAP.MOD[{PROD}].WELL[{A1}].File" } }, { "file": {"id": 1333123123}, "arguments": { "address": "GAP.MOD[{PROD}].WELL[{A1}].VLPFile" } } ] } ``` * The first dependency maps a PROSPER `.OUT` file for well A1 * The second dependency maps a PROSPER `.VLP` file for the same well * The `address` field uses GAP's path notation to specify where each file should be linked in the model ## Run as a Windows service Download the *gap-setup-\{VERSION}.exe* file via the download links available from Data fusion > Integrate > **Extractors** in CDF. Run the installer on a Windows operating system. A wizard guides you through the installation process. Select the destination folder where the connector will be installed. By default, this is **C:\Cognite\\**. Inside this folder, the **GapConnector** working directory is created. The connector expects a `config.yml` file as the configuration in the same directory. You'll find templates for the configuration file in the `config` folder. Open the **Services** app in Windows and find the **Cognite GAP Connector** service. Right-click the service and select **Start**. ## Uninstall and update the connector Run the same MSI installer to uninstall the connector. The installer deletes installed binaries, removes the service, and cleans the application registry. The installation is not deleted and still contains the configuration file, logs, and application data generated by the connector. If the connector is re-installed, it resumes the operation using the saved state. When a new version of the GAP connector is available, run the MSI installer for the new version to upgrade the connector to the new version. After the upgrade, you must manually restart the service. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_troubleshooting Find solutions to common GAP connector issues including authentication errors, license problems, and file read failures. ## The connector receives 401 errors from CDF [HTTP error 401](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401) means that the service didn't recognize the authentication provided by the client. If the Cognite API returns 401 errors to the connector, it means the connector successfully received an authentication token, but the token isn't valid for the project. This usually occurs because the `project` or `host` parameters in the `cognite` section are incorrect. ## The connector is not setting the data set when using remote configuration If you set up the connector to run with remote configuration, only the `host`, `project`, `idp-authentication`, and `extraction-pipeline` parameters in the `cognite` section are read from the file on the local machine. Other parameters in the `cognite` section, such as `data-set`, must be added to the configuration file in the extraction pipelines. ## Diagnose license-related issues The connector will fail to run properly if you don't have a PETEX OpenServer license. The following PowerShell commands can be used to verify if PETEX OpenServer and GAP licenses are available in the environment where the connector is running. ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Create a new OpenServer COM object $server = New-Object -ComObject "PX32.OpenServer.1" # Test if a GAP license is available $server.DoCommand("GAP.START()") # Close GAP $server.DoCommand("GAP.SHUTDOWN") # Release the COM object [System.Runtime.Interopservices.Marshal]::ReleaseComObject($server) | Out-Null [System.GC]::Collect() [System.GC]::WaitForPendingFinalizers() ``` ## File-read issues in model extractions The error `"File Read Failed (3160)"` typically occurs when a model file can't be opened within the installed version of IPM. This error is typically due to a version mismatch. If the model file was saved with a newer version of the tool, for instance IPM 13.5, and opened using OpenServer in an older version, for instance IPM 13.0, the version mismatch prevents the file from being read. Projects in IPM Suite aren't backward-compatible due to structure amendments and new features. # Unit conversion Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/gap/gap_units Learn how the GAP connector handles unit conversion with supported quantities, units, and conversion factors. The GAP connector converts units internally based upon conversion factors. This functionality is implemented in the connector since OpenServer, the protocol used to integrate Cognite Data Fusion (CDF) with the GAP simulator, doesn't provide the ability for arbitrary unit conversion. When you configure a simulation routine revision, you can specify units for reading values from the simulator and units for setting values from CDF. If you don't specify a unit, values are transferred without conversion. Unit conversion can only be used for the variables types `DOUBLE` and `DOUBLE_ARRAY`. The conversion logic follows this formula: vtarget = (vsource - s) × m Where: | Symbol | Description | | - | - | | vtarget | The resulting value in the target unit. | | vsource | The original value in the source unit. | | s | The conversion shift factor. | | m | The conversion multiplier factor. | The conversion process follows these steps: Verify that the specified quantity exists in the unit conversion database. Confirm that the desired unit exists for the selected quantity. Retrieve the unit associated with the OpenServer address using the `UNITNAME` extension. Verify its existence in the unit conversion database for the selected quantity. Convert the units for both scalar and array values. * If you specify an invalid quantity or unit in your simulator routine, the connector will fail the simulation run. * The table below may not be comprehensive for new simulator releases. Customers can request the addition of new units as needed. This table lists all supported quantities, units, and their associated conversion factors from the GAP simulator unit management system: | Quantity | Unit | Multiplier | Shift | | - | - | - | - | | Acceleration | ft/sec/sec | 1.0 | 0.0 | | Acceleration | m/sec/sec | 0.3048 | 0.0 | | Adsorption Diffusion Constant | 1/day | 1.0 | 0.0 | | Adsorption Diffusion Constant | 1/month | 30.5 | 0.0 | | Adsorption Diffusion Constant | 1/year | 365.25 | 0.0 | | Adsorption Diffusion Constant | 1/sec | 1.157407407407e-05 | 0.0 | | Angle | degrees | 1.0 | 0.0 | | Angle | radians | 0.0174533 | 0.0 | | Anisotropy | fraction | 1.0 | 0.0 | | Anisotropy | percent | 100.0 | 0.0 | | BTU Cost | US\$/MMBTU | 1.0 | 0.0 | | Bean Size | inches | 1.0 | 0.0 | | Bean Size | m | 0.0254 | 0.0 | | Bean Size | feet | 0.08333333 | 0.0 | | Bean Size | cm | 2.54 | 0.0 | | Bean Size | mm | 25.4 | 0.0 | | Bean Size | km | 2.54e-05 | 0.0 | | Bean Size | miles | 1.58e-05 | 0.0 | | Bean Size | 64ths inch | 64.0 | 0.0 | | Blade Height | mm | 1.0 | 0.0 | | Blade Height | ins | 25.4 | 0.0 | | Blade Height | cm | 10.0 | 0.0 | | Blade Height | 64ths | 0.396875 | 0.0 | | Boiling Temperature | deg F | 1.0 | 0.0 | | Boiling Temperature | deg C | 0.5555555 | -32.0 | | Boiling Temperature | deg R | 1.0 | 459.67 | | Boiling Temperature | deg K | 0.5555555 | 459.67 | | C (Gas) | Mscf/day/psi2 | 1.0 | 0.0 | | C (Gas) | Sm3/day/bar2 | 28.17399 | 0.0 | | C (Gas) | m3/day/kPa2 | 28.173 | 0.0 | | C (Gas) | m3Vn/h/bar2 | 1.11277 | 0.0 | | C (Oil) | STB/day/psi2 | 1.0 | 0.0 | | C (Oil) | Sm3/day/bar2 | 0.1589873 | 0.0 | | C (Oil) | m3/day/kPa2 | 0.1589873 | 0.0 | | C (Oil) | m3/hour/bar2 | 0.006625 | 0.0 | | CGR / WGR | STB/MMscf | 1.0 | 0.0 | | CGR / WGR | Sm3/Sm3 | 5.64e-06 | 0.0 | | CGR / WGR | m3/m3 | 5.64e-06 | 0.0 | | CGR / WGR | m3/m3Vn | 5.95e-06 | 0.0 | | CGR / WGR | STB/Mscf | 0.001 | 0.0 | | CGR / WGR | Sm3/kSm3 | 0.0056433 | 0.0 | | CGR / WGR | Sm3/MSm3 | 5.6433 | 0.0 | | Calendar Date | date | 1.0 | 0.0 | | Casing Weight | lb/ft | 1.0 | 0.0 | | Casing Weight | Kg/m | 1.488164 | 0.0 | | Casing Yield | Kpsi | 1.0 | 0.0 | | Casing Yield | Kbar | 0.06894757 | 0.0 | | Casing Yield | Pa | 68947570.0 | 0.0 | | Casing Yield | Kg/cm2 | 70.31 | 0.0 | | Casing Yield | psi | 1000.0 | 0.0 | | Casing Yield | bar | 68.94757 | 0.0 | | Choke Area | in2 | 1.0 | 0.0 | | Choke Area | cm2 | 6.45161 | 0.0 | | Choke Area | ft2 | 0.00694444 | 0.0 | | Choke Area | m2 | 0.000645161 | 0.0 | | Compressibility | 1/psi | 1.0 | 0.0 | | Compressibility | 1/bar | 14.503774 | 0.0 | | Compressibility | 1/Kpa | 0.14503774 | 0.0 | | Compressibility | 1/(Kg/cm2) | 14.22273 | 0.0 | | Conductivity | BTU/hr/ft/F | 1.0 | 0.0 | | Conductivity | W/m/K | 1.730735 | 0.0 | | Critical Pressure | psig | 1.0 | 0.0 | | Critical Pressure | psia | 1.0 | 14.696 | | Critical Pressure | BARa | 0.06894757 | 14.696 | | Critical Pressure | kPa a | 6.894757 | 14.696 | | Critical Pressure | kPa g | 6.894757 | 0.0 | | Critical Pressure | BARg | 0.06894757 | 0.0 | | Critical Pressure | Kg/cm2 a | 0.07031 | 14.696 | | Critical Pressure | Kg/cm2 g | 0.07031 | 0.0 | | Critical Pressure | atm a | 0.068046 | 14.696 | | Critical Temperature | deg F | 1.0 | 0.0 | | Critical Temperature | deg C | 0.5555555 | -32.0 | | Critical Temperature | deg R | 1.0 | 459.67 | | Critical Temperature | deg K | 0.5555555 | 459.67 | | Critical Volume | ft3/lb.mole | 1.0 | 0.0 | | Critical Volume | l/kg.mole | 62.4297 | 0.0 | | Critical Volume | m3/kg.mole | 0.06243 | 0.0 | | Cumulative Flow | ft3 | 1.0 | 0.0 | | Cumulative Flow | m3 | 0.02832 | 0.0 | | Currency | MMUS\$ | 1.0 | 0.0 | | Currency | US \$ | 1000000.0 | 0.0 | | Current | amps | 1.0 | 0.0 | | Daily Currency | MMUS\$/day | 1.0 | 0.0 | | Damaged Zone Thickness | inches | 1.0 | 0.0 | | Damaged Zone Thickness | m | 0.0254 | 0.0 | | Damaged Zone Thickness | feet | 0.08333333 | 0.0 | | Damaged Zone Thickness | cm | 2.54 | 0.0 | | Damaged Zone Thickness | mm | 25.4 | 0.0 | | Damaged Zone Thickness | km | 2.54e-05 | 0.0 | | Damaged Zone Thickness | miles | 1.58e-05 | 0.0 | | Damaged Zone Thickness | 64ths inch | 64.0 | 0.0 | | Darcy Coefficient (Gas) | psi2/(Mscf/day) | 1.0 | 0.0 | | Darcy Coefficient (Gas) | bar2/(Sm3/day) | 0.0001685 | 0.0 | | Darcy Coefficient (Gas) | kPa2/(m3/day) | 1.687 | 0.0 | | Darcy Coefficient (Gas) | bar2/(m3\[Vn]/h) | 0.004272 | 0.0 | | Darcy Coefficient (Gas) | Kg/cm2/(Sm3/d) | 0.0001718296 | 0.0 | | Darcy Coefficient (Gas) | bar2/(MSm3/day) | 168.5 | 0.0 | | Darcy Coefficient (Oil) | psi/(STB/day) | 1.0 | 0.0 | | Darcy Coefficient (Oil) | bar/(Sm3/day) | 0.433667 | 0.0 | | Darcy Coefficient (Oil) | kPa/(m3/day) | 43.366715 | 0.0 | | Darcy Coefficient (Oil) | bar/(m3\[Vn]/h) | 10.408012 | 0.0 | | Darcy Coefficient (Oil) | kg/cm2/(Sm3/day) | 0.442217 | 0.0 | | Darcy Coefficient (Pseudo) | psi2/cp/(Mscf/day) | 1.0 | 0.0 | | Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(Sm3/day) | 0.1685 | 0.0 | | Darcy Coefficient (Pseudo) | kPa2/(Pa.s)/(m3/day) | 1687.0 | 0.0 | | Darcy Coefficient (Pseudo) | bar2/(Pa.s)/m3Vn/h) | 4.272 | 0.0 | | Darcy Coefficient (Pseudo) | (Kg/cm2)2/(Pa.s)/(Sm3/d) | 0.175225 | 0.0 | | Date | day | 1.0 | 0.0 | | Date | week | 0.1428571 | 0.0 | | Date | month | 0.03285421 | 0.0 | | Date | year | 0.0027378 | 0.0 | | Date | date/time | 1.0 | 0.0 | | Date | date | 1.0 | 0.0 | | Date | hour | 24.0 | 0.0 | | Date | minute | 1440.0 | 0.0 | | Date | second | 86400.0 | 0.0 | | Debris Weight | g | 1.0 | 0.0 | | Debris Weight | Kg | 0.001 | 0.0 | | Density | lb/ft3 | 1.0 | 0.0 | | Density | Kg/m3 | 16.0185 | 0.0 | | Density | g/cc | 0.0160185 | 0.0 | | Density | g/L | 16.0185 | 0.0 | | Density (g/L) | g/L | 1.0 | 0.0 | | Density (g/L) | lb/ft3 | 0.06242797 | 0.0 | | Density (g/L) | g/cc | 0.001 | 0.0 | | Density (g/L) | kg/m3 | 1.0 | 0.0 | | Density (g/L) | kg/L | 0.001 | 0.0 | | Diameter | inches | 1.0 | 0.0 | | Diameter | m | 0.0254 | 0.0 | | Diameter | feet | 0.08333333 | 0.0 | | Diameter | cm | 2.54 | 0.0 | | Diameter | mm | 25.4 | 0.0 | | Diameter | km | 2.54e-05 | 0.0 | | Diameter | miles | 1.58e-05 | 0.0 | | Diameter | 64ths inch | 64.0 | 0.0 | | Drainage Area | acres | 1.0 | 0.0 | | Drainage Area | m2 | 4046.9 | 0.0 | | Drainage Area | hectares | 0.40469 | 0.0 | | Drainage Area | ft2 | 43560.0 | 0.0 | | EOS Gas Volume | scf | 1.0 | 0.0 | | EOS Gas Volume | Sm3 | 0.0283168 | 0.0 | | EOS Mass | lbm | 1.0 | 0.0 | | EOS Mass | Kg | 0.45359237 | 0.0 | | EOS Mass | s. ton | 0.0005 | 0.0 | | EOS Mass | l. ton | 0.0004464285714 | 0.0 | | EOS Mass | tonne | 0.00045359237 | 0.0 | | EOS Oil Volume | STB | 1.0 | 0.0 | | EOS Oil Volume | Sm3 | 0.158987 | 0.0 | | EOS Oil Volume | scf | 5.615 | 0.0 | | Elastic Constant | in/lb/ft | 1.0 | 0.0 | | Elastic Constant | cm/kg/m | 18.3715 | 0.0 | | Emissivity | fraction | 1.0 | 0.0 | | Emissivity | percent | 100.0 | 0.0 | | Energy | BTU | 1.0 | 0.0 | | Energy | W-h | 0.293064 | 0.0 | | Energy | kW-h | 0.000293064 | 0.0 | | Energy | MMBTU | 1e-06 | 0.0 | | Energy | MW-h | 2.93064e-07 | 0.0 | | Energy | BBTU | 1e-09 | 0.0 | | Energy | GW-h | 2.93064e-10 | 0.0 | | Energy | TBTU | 1e-12 | 0.0 | | Energy | TW-h | 2.93064e-13 | 0.0 | | Energy | J | 1055.0304 | 0.0 | | Energy | kJ | 1.0550304 | 0.0 | | Energy per unit mass | J/Kg | 1.0 | 0.0 | | Energy per unit mass | KJ/Kg | 0.001 | 0.0 | | Energy per unit mass | BTU/lb | 0.4299226 | 0.0 | | Enthalpy | BTU/lb | 1.0 | 0.0 | | Enthalpy | J/gm | 2.326 | 0.0 | | Enthalpy | KJ/kg | 2.326 | 0.0 | | Entropy | BTU/lb.R | 1.0 | 0.0 | | Entropy | J/g.K | 4.1868 | 0.0 | | Entropy | KJ/Kg.K | 4.1868 | 0.0 | | Erosion | 0.001 inches/year | 1.0 | 0.0 | | Erosion | mm/year | 0.0254 | 0.0 | | Explosive Weight | g | 1.0 | 0.0 | | Explosive Weight | Kg | 0.001 | 0.0 | | Flow Duration | hours | 1.0 | 0.0 | | Flow Duration | minutes | 60.0 | 0.0 | | Flow Duration | seconds | 360.0 | 0.0 | | Flow Duration | days | 0.0416667 | 0.0 | | Fluid Gravity | sp. gravity | 1.0 | 0.0 | | Fluid Loss Time | minutes | 1.0 | 0.0 | | Fluid Loss Time | seconds | 60.0 | 0.0 | | Fluid Loss Time | hours | 0.016666667 | 0.0 | | Fluid Loss Time | days | 0.00069444444 | 0.0 | | Force | lbf | 1.0 | 0.0 | | Force | N | 4.448221 | 0.0 | | Fraction | fraction | 1.0 | 0.0 | | Fraction | percent | 100.0 | 0.0 | | Fracture Half Length | feet | 1.0 | 0.0 | | Fracture Half Length | m | 0.3048 | 0.0 | | Fracture Half Length | miles | 0.000189393 | 0.0 | | Fracture Half Length | km | 0.0003048 | 0.0 | | Fracture Half Length | inches | 12.0 | 0.0 | | Fracture Half Length | cm | 30.48 | 0.0 | | Fracture Half Length | mm | 304.8 | 0.0 | | Fracture Half Length | 64ths inch | 768.0 | 0.0 | | Fracture Height | feet | 1.0 | 0.0 | | Fracture Height | m | 0.3048 | 0.0 | | Fracture Height | miles | 0.000189393 | 0.0 | | Fracture Height | km | 0.0003048 | 0.0 | | Fracture Height | inches | 12.0 | 0.0 | | Fracture Height | cm | 30.48 | 0.0 | | Fracture Height | mm | 304.8 | 0.0 | | Fracture Height | 64ths inch | 768.0 | 0.0 | | Frequency | Hertz | 1.0 | 0.0 | | Frequency | kHz | 0.001 | 0.0 | | GOR / GLR | scf/STB | 1.0 | 0.0 | | GOR / GLR | Sm3/Sm3 | 0.1772 | 0.0 | | GOR / GLR | m3/m3 | 0.1772 | 0.0 | | GOR / GLR | m3Vn/m3 | 0.168 | 0.0 | | GOR / GLR | Mscf/STB | 0.001 | 0.0 | | GOR / GLR | kSm3/Sm3 | 0.0001772 | 0.0 | | Gas Cost | US\$/MMscf | 1.0 | 0.0 | | Gas Cost | US\$/1000Sm3 | 0.0355 | 0.0 | | Gas Cost | US\$/1000m3 | 0.0355 | 0.0 | | Gas Cost | US\$/1000\[Vn]m3 | 0.0355 | 0.0 | | Gas FVF | ft3/scf | 1.0 | 0.0 | | Gas FVF | m3/Sm3 | 1.005071 | 0.0 | | Gas FVF | m3/m3 (st) | 1.005071 | 0.0 | | Gas FVF | m3/m3 Vn | 1.060295 | 0.0 | | Gas FVF | RB/mcf | 178.0944 | 0.0 | | Gas FVF | RB/scf | 0.1780944 | 0.0 | | Gas FVF | RB/Mscf | 178.0944 | 0.0 | | Gas Gravity | sp. gravity | 1.0 | 0.0 | | Gas Gravity | lb/ft3 | 0.0764 | 0.0 | | Gas Gravity | kg/m3 | 1.22381 | 0.0 | | Gas Gravity | g/cc | 0.00122381 | 0.0 | | Gas Injection | MMscf | 1.0 | 0.0 | | Gas Injection | MSm3 | 0.028174 | 0.0 | | Gas Injection | Bscf | 0.001 | 0.0 | | Gas Injection | BSm3 | 2.8174e-05 | 0.0 | | Gas Injection | GSm3 | 2.8174e-05 | 0.0 | | Gas Injection | Mscf | 1000.0 | 0.0 | | Gas Injection | kSm3 | 28.17399 | 0.0 | | Gas Injection | Mm3Vn | 0.02670659 | 0.0 | | Gas Injection | km3Vn | 26.70659 | 0.0 | | Gas Injection | Gm3Vn | 2.670659e-05 | 0.0 | | Gas Injection | scf | 1000000.0 | 0.0 | | Gas Injection | Sm3 | 28174.0 | 0.0 | | Gas Injection | m3Vn | 26706.59 | 0.0 | | Gas Production | MMscf | 1.0 | 0.0 | | Gas Production | MSm3 | 0.028174 | 0.0 | | Gas Production | Bscf | 0.001 | 0.0 | | Gas Production | BSm3 | 2.8174e-05 | 0.0 | | Gas Production | GSm3 | 2.8174e-05 | 0.0 | | Gas Production | Mscf | 1000.0 | 0.0 | | Gas Production | kSm3 | 28.17399 | 0.0 | | Gas Production | Mm3Vn | 0.02670659 | 0.0 | | Gas Production | km3Vn | 26.70659 | 0.0 | | Gas Production | Gm3Vn | 2.670659e-05 | 0.0 | | Gas Production | scf | 1000000.0 | 0.0 | | Gas Production | Sm3 | 28174.0 | 0.0 | | Gas Production | m3Vn | 26706.59 | 0.0 | | Gas Rate | MMscf/day | 1.0 | 0.0 | | Gas Rate | 1000Sm3/d | 28.17399 | 0.0 | | Gas Rate | 1000m3/d | 28.17399 | 0.0 | | Gas Rate | m3\[Vn]/h | 1112.774 | 0.0 | | Gas Rate | Mscf/day | 1000.0 | 0.0 | | Gas Rate | scf/day | 1000000.0 | 0.0 | | Gas Rate | Sm3/day | 28173.99 | 0.0 | | Gas Rate per Unit Length | MMscf/day/ft | 1.0 | 0.0 | | Gas Rate per Unit Length | 1000Sm3/d/m | 92.43435 | 0.0 | | Gas Volume | MMscf | 1.0 | 0.0 | | Gas Volume | MSm3 | 0.028174 | 0.0 | | Gas Volume | Bscf | 0.001 | 0.0 | | Gas Volume | BSm3 | 2.8174e-05 | 0.0 | | Gas Volume | GSm3 | 2.8174e-05 | 0.0 | | Gas Volume | Mscf | 1000.0 | 0.0 | | Gas Volume | kSm3 | 28.17399 | 0.0 | | Gas Volume | Mm3Vn | 0.02670659 | 0.0 | | Gas Volume | km3Vn | 26.70659 | 0.0 | | Gas Volume | Gm3Vn | 2.670659e-05 | 0.0 | | Gas Volume | scf | 1000000.0 | 0.0 | | Gas Volume | Sm3 | 28174.0 | 0.0 | | Gas Volume | m3Vn | 26706.59 | 0.0 | | Grain Diameter | inches | 1.0 | 0.0 | | Grain Diameter | m | 0.0254 | 0.0 | | Grain Diameter | feet | 0.08333333 | 0.0 | | Grain Diameter | cm | 2.54 | 0.0 | | Grain Diameter | mm | 25.4 | 0.0 | | Grain Diameter | km | 2.54e-05 | 0.0 | | Grain Diameter | miles | 1.58e-05 | 0.0 | | Grain Diameter | 64ths inch | 64.0 | 0.0 | | Gravel Pack Length | inches | 1.0 | 0.0 | | Gravel Pack Length | m | 0.0254 | 0.0 | | Gravel Pack Length | feet | 0.08333333 | 0.0 | | Gravel Pack Length | cm | 2.54 | 0.0 | | Gravel Pack Length | mm | 25.4 | 0.0 | | Gravel Pack Length | km | 2.54e-05 | 0.0 | | Gravel Pack Length | miles | 1.58e-05 | 0.0 | | Gravel Pack Length | 64ths inch | 64.0 | 0.0 | | Head | feet | 1.0 | 0.0 | | Head | m | 0.3048 | 0.0 | | Head | miles | 0.000189393 | 0.0 | | Head | km | 0.0003048 | 0.0 | | Head | inches | 12.0 | 0.0 | | Head | cm | 30.48 | 0.0 | | Head | mm | 304.8 | 0.0 | | Head | 64ths inch | 768.0 | 0.0 | | Heat Capacity Rate | BTU/f/s | 1.0 | 0.0 | | Heat Capacity Rate | cal/C/s | 453.5924 | 0.0 | | Heat Capacity Rate | J/K/s | 1899.101 | 0.0 | | Heat Capacity Rate | Kg.m2/s3/K | 1899.101 | 0.0 | | Heat Duty | BTU/d | 1.0 | 0.0 | | Heat Duty | W | 0.012211 | 0.0 | | Heat Duty | MMBTU/d | 1e-06 | 0.0 | | Heat Duty | MW | 1.2211e-08 | 0.0 | | Heat Duty | BBTU/d | 1e-09 | 0.0 | | Heat Duty | GW | 1.2211e-11 | 0.0 | | Heat Duty | TBTU/d | 1e-12 | 0.0 | | Heat Duty | TW | 1.2211e-14 | 0.0 | | Heat Duty | kW | 1.2211e-05 | 0.0 | | Heat Transfer Coefficient | BTU/h/ft2/F | 1.0 | 0.0 | | Heat Transfer Coefficient | W/m2/K | 5.678983 | 0.0 | | Heat Transfer Coefficient | kJ/m2/h/K | 20.44175 | 0.0 | | Heat Transfer per deg F | BTU/h/F | 1.0 | 0.0 | | Heat Transfer per deg F | W/degC | 0.527059 | 0.0 | | Heat Transfer per deg F | W/K | 0.527059 | 0.0 | | Heating Value | BTU/scf | 1.0 | 0.0 | | Heating Value | kJ/sm3 | 37.25894617 | 0.0 | | Height Difference | inches | 1.0 | 0.0 | | Height Difference | m | 0.0254 | 0.0 | | Height Difference | feet | 0.08333333 | 0.0 | | Height Difference | cm | 2.54 | 0.0 | | Height Difference | mm | 25.4 | 0.0 | | Height Difference | km | 2.54e-05 | 0.0 | | Height Difference | miles | 1.58e-05 | 0.0 | | Height Difference | 64ths inch | 64.0 | 0.0 | | Humidity | percent | 1.0 | 0.0 | | Humidity | fraction | 0.01 | 0.0 | | Injected Gas Rate | Mscf/day | 1.0 | 0.0 | | Injected Gas Rate | scf/day | 1000.0 | 0.0 | | Injected Gas Rate | Sm3/day | 28.17399 | 0.0 | | Injected Gas Rate | m3\[Vn]/h | 1.1127743 | 0.0 | | Interfacial Tension | dyne/cm | 1.0 | 0.0 | | Interfacial Tension | mN/m | 1.0 | 0.0 | | Interfacial Tension | N/m | 0.001 | 0.0 | | Inverse Liquid Rate | day/STB | 1.0 | 0.0 | | Inverse Liquid Rate | day/Sm3 | 6.28981057 | 0.0 | | Inverse Liquid Rate | day/m3 | 6.28981057 | 0.0 | | Inverse Liquid Rate | hour/m3 | 150.94339 | 0.0 | | Inverse Liquid Rate | day/scf | 0.17809439 | 0.0 | | Joules Thomson Coefficient | degrees F/psi | 1.0 | 0.0 | | Joules Thomson Coefficient | degrees K/bar | 7.80556 | 0.0 | | Kinematic viscosity (n) | .e-6 m2/s | 1.0 | 0.0 | | Kinematic viscosity (n) | .e-6 ft2/s | 10.76391 | 0.0 | | Kinematic viscosity (n) | cSt | 1.0 | 0.0 | | Langmuir Pressure | psig | 1.0 | 0.0 | | Langmuir Pressure | psia | 1.0 | 14.696 | | Langmuir Pressure | BARa | 0.06894757 | 14.696 | | Langmuir Pressure | kPa a | 6.894757 | 14.696 | | Langmuir Pressure | kPa g | 6.894757 | 0.0 | | Langmuir Pressure | BARg | 0.06894757 | 0.0 | | Langmuir Pressure | Kg/cm2 a | 0.07031 | 14.696 | | Langmuir Pressure | Kg/cm2 g | 0.07031 | 0.0 | | Langmuir Pressure | atm a | 0.068046 | 14.696 | | Langmuir Volume Constant (/mass) | scf/(s. ton) | 1.0 | 0.0 | | Langmuir Volume Constant (/mass) | scf/(l. ton) | 1.12 | 0.0 | | Langmuir Volume Constant (/mass) | scf/tonne | 1.102311 | 0.0 | | Langmuir Volume Constant (/mass) | Sm3/tonne | 0.031214 | 0.0 | | Langmuir Volume Constant (/mass) | Sm3/kg | 1.560707e-05 | 0.0 | | Langmuir Volume Constant (/volume) | scf/ft3 | 1.0 | 0.0 | | Langmuir Volume Constant (/volume) | Sm3/m3 | 1.0 | 0.0 | | Length | feet | 1.0 | 0.0 | | Length | m | 0.3048 | 0.0 | | Length | miles | 0.000189393 | 0.0 | | Length | km | 0.0003048 | 0.0 | | Length | inches | 12.0 | 0.0 | | Length | cm | 30.48 | 0.0 | | Length | mm | 304.8 | 0.0 | | Length | 64ths inch | 768.0 | 0.0 | | Length (GAP) | inches | 1.0 | 0.0 | | Length (GAP) | m | 0.0254 | 0.0 | | Length (GAP) | feet | 0.08333333 | 0.0 | | Length (GAP) | cm | 2.54 | 0.0 | | Length (GAP) | mm | 25.4 | 0.0 | | Length (GAP) | km | 2.54e-05 | 0.0 | | Length (GAP) | miles | 1.58e-05 | 0.0 | | Length (GAP) | 64ths inch | 64.0 | 0.0 | | Linear Density | lb/ft | 1.0 | 0.0 | | Linear Density | Kg/m | 1.488164 | 0.0 | | Liquid Dropout | percent | 1.0 | 0.0 | | Liquid Dropout | fraction | 0.01 | 0.0 | | Liquid Rate | STB/day | 1.0 | 0.0 | | Liquid Rate | Sm3/day | 0.1589873 | 0.0 | | Liquid Rate | m3/day | 0.1589873 | 0.0 | | Liquid Rate | m3/hour | 0.006625 | 0.0 | | Liquid Rate | scf/day | 5.616 | 0.0 | | Liquid Rate (Reservoir) | RB/day | 1.0 | 0.0 | | Liquid Rate (Reservoir) | m3/day | 0.159 | 0.0 | | Liquid Rate (Reservoir) | m3/hour | 0.006625 | 0.0 | | Liquid Rate Per HP | STB/day/hp | 1.0 | 0.0 | | Liquid Rate Per HP | Sm3/day/kw | 0.11855683 | 0.0 | | Liquid Rate Per HP | m3/day/kw | 0.11855683 | 0.0 | | Liquid Rate Per HP | m3/hour/kw | 0.004940263 | 0.0 | | Liquid Rate Per HP | scf/day/hp | 5.615 | 0.0 | | Liquid Rate per Unit Length | STB/day/ft | 1.0 | 0.0 | | Liquid Rate per Unit Length | Sm3/day/m | 0.5216119 | 0.0 | | Liquid Reservoir Rates | RB/day | 1.0 | 0.0 | | Liquid Reservoir Rates | Rm3/day | 0.159 | 0.0 | | Liquid Reservoir Rates | RMMcf/day | 5.615e-06 | 0.0 | | Liquid Reservoir Rates | Rm3/hour | 0.006625 | 0.0 | | Liquid Volume | MMSTB | 1.0 | 0.0 | | Liquid Volume | MSm3 | 0.159 | 0.0 | | Liquid Volume | BSTB | 0.001 | 0.0 | | Liquid Volume | BSm3 | 0.000159 | 0.0 | | Liquid Volume | MSTB | 1000.0 | 0.0 | | Liquid Volume | kSm3 | 159.0 | 0.0 | | Liquid Volume | Mm3 | 0.159 | 0.0 | | Liquid Volume | km3 | 159.0 | 0.0 | | Liquid Volume | GSm3 | 0.000159 | 0.0 | | Liquid Volume | Gm3 | 0.000159 | 0.0 | | Liquid Volume | STB | 1000000.0 | 0.0 | | Liquid Volume | Sm3 | 159000.0 | 0.0 | | Liquid Volume | m3 | 159000.0 | 0.0 | | Liquid Volume (STB) | STB | 1.0 | 0.0 | | Liquid Volume (STB) | Sm3 | 0.158987 | 0.0 | | Liquid Volume (STB) | scf | 5.615 | 0.0 | | Liquid produced (res cond) | MMRB | 1.0 | 0.0 | | Liquid produced (res cond) | MRm3 | 0.159 | 0.0 | | Liquid produced (res cond) | BRB | 0.001 | 0.0 | | Liquid produced (res cond) | BRm3 | 0.000159 | 0.0 | | Log Permeability | md | 1.0 | 0.0 | | Log Permeability | um2 | 0.0009869 | 0.0 | | Log Permeability | darcy | 0.001 | 0.0 | | Mass | lbm | 1.0 | 0.0 | | Mass | Kg | 0.45359237 | 0.0 | | Mass | s. ton | 0.0005 | 0.0 | | Mass | l. ton | 0.0004464285714 | 0.0 | | Mass | tonne | 0.00045359237 | 0.0 | | Mass Concentration | kg/kg | 1.0 | 0.0 | | Mass Concentration | ppm | 1000000.0 | 0.0 | | Mass Concentration | mg/g | 1000.0 | 0.0 | | Mass Flow Rate | lbm/day | 1.0 | 0.0 | | Mass Flow Rate | Kg/day | 0.45359237 | 0.0 | | Mass Flow Rate | lbm/sec | 1.16e-05 | 0.0 | | Mass Flow Rate | s. ton/day | 0.0005 | 0.0 | | Mass Flow Rate | l. ton/day | 0.0004464285714 | 0.0 | | Mass Flow Rate | tonne/day | 0.00045359237 | 0.0 | | Mass Flow Rate | Kg/sec | 5.25e-06 | 0.0 | | Mass Flow Rate | lbm/hr | 0.04166666667 | 0.0 | | Mass Flow Rate | Kg/hr | 0.01889968208 | 0.0 | | Molar Flow Rate | lb-moles/day | 1.0 | 0.0 | | Molar Flow Rate | kg-moles/day | 0.45359237 | 0.0 | | Mole Composition | percent | 1.0 | 0.0 | | Mole Composition | fraction | 0.01 | 0.0 | | Molecular Weight | lb/lb.mole | 1.0 | 0.0 | | Molecular Weight | g/g.mole | 1.0 | 0.0 | | Molecular Weight | kg/kg.mole | 1.0 | 0.0 | | Motor Speed | rpm | 1.0 | 0.0 | | Motor Speed | rps | 0.0166667 | 0.0 | | Motor Speed | Hertz | 0.0166667 | 0.0 | | Mud Density | lb/US gal | 1.0 | 0.0 | | Mud Density | lb/ft3 | 7.4805225 | 0.0 | | Mud Density | Kg/m3 | 119.8264 | 0.0 | | Non-Darcy Coefficient (Gas) | psi2/(Mscf/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(Sm3/day)2 | 5.98e-06 | 0.0 | | Non-Darcy Coefficient (Gas) | kPa2/(m3/day)2 | 0.05589 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(m3\[Vn]/h)2 | 0.003839 | 0.0 | | Non-Darcy Coefficient (Gas) | (Kg/cm2)2/(Sm3/d)2 | 6.22e-06 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(MSm3/day)2 | 5980000.0 | 0.0 | | Non-Darcy Coefficient (Oil) | psi/(STB/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Oil) | bar/(Sm3/day)2 | 2.727684 | 0.0 | | Non-Darcy Coefficient (Oil) | kPa/(m3/day)2 | 272.7684 | 0.0 | | Non-Darcy Coefficient (Oil) | bar/(m3\[Vn]/h)2 | 1571.15 | 0.0 | | Non-Darcy Coefficient (Oil) | Kg/cm2/(Sm3/day)2 | 2.781464 | 0.0 | | Non-Darcy Coefficient (Pseudo) | psi2/cp/(Mscf/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(Sm3/day)2 | 0.00598 | 0.0 | | Non-Darcy Coefficient (Pseudo) | kPa2/(Pa.s)/(m3/day)2 | 55.89 | 0.0 | | Non-Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(m3Vn/h)2 | 3.839 | 0.0 | | Non-Darcy Coefficient (Pseudo) | (kg/cm2)2/(Pa.s)/(Sm3/d)2 | 0.006218669 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(Mscf/day) | 1.0 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(Sm3/d) | 0.0354937 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(m3/d) | 0.0354937 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(m3\[Vn]/h) | 0.0008986551 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(STB/day) | 1.0 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(Sm3/d) | 6.2898106 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(m3/d) | 6.2898106 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(m3/hr) | 150.9434 | 0.0 | | Oil Cost | US\$/STB | 1.0 | 0.0 | | Oil Cost | US\$/Sm3 | 6.289 | 0.0 | | Oil Cost | US\$/m3 | 6.289 | 0.0 | | Oil Cost | US\$/\[Vn]m3 | 6.289 | 0.0 | | Oil FVF | RB/STB | 1.0 | 0.0 | | Oil FVF | m3/Sm3 | 1.0 | 0.0 | | Oil FVF | m3/m3 | 1.0 | 0.0 | | Oil Gravity | API | 1.0 | 0.0 | | Oil Gravity | rel.density | 141.5 | 131.5 | | Oil Gravity | Kg/m3 | 141500.0 | 131.5 | | Oil Production | MMSTB | 1.0 | 0.0 | | Oil Production | MSm3 | 0.159 | 0.0 | | Oil Production | BSTB | 0.001 | 0.0 | | Oil Production | BSm3 | 0.000159 | 0.0 | | Oil Production | MSTB | 1000.0 | 0.0 | | Oil Production | kSm3 | 159.0 | 0.0 | | Oil Production | Mm3 | 0.159 | 0.0 | | Oil Production | km3 | 159.0 | 0.0 | | Oil Production | GSm3 | 0.000159 | 0.0 | | Oil Production | Gm3 | 0.000159 | 0.0 | | Oil Production | STB | 1000000.0 | 0.0 | | Oil Production | Sm3 | 159000.0 | 0.0 | | Oil Production | m3 | 159000.0 | 0.0 | | OneSubsea Pump Rate | bbl/d | 1.0 | 0.0 | | OneSubsea Pump Rate | Am3/h | 0.006625 | 0.0 | | OneSubsea Pump Speed | rpm | 1.0 | 0.0 | | OneSubsea Pump Speed | HZ | 0.0166667 | 0.0 | | OneSubsea Pump dP | psi | 1.0 | 0.0 | | OneSubsea Pump dP | bar | 0.06894757 | 0.0 | | OneSubsea Pump dP | kPa | 6.894757 | 0.0 | | OneSubsea Pump dP | Kg/cm2 | 0.07031 | 0.0 | | OneSubsea Pump dP | Pa | 6894.757 | 0.0 | | Pad Volume | bbl | 1.0 | 0.0 | | Pad Volume | m3 | 0.1589873 | 0.0 | | Pad Volume | ft3 | 5.615 | 0.0 | | Penetration | fraction | 1.0 | 0.0 | | Penetration | percent | 100.0 | 0.0 | | Percent | percent | 1.0 | 0.0 | | Percent | fraction | 0.01 | 0.0 | | Perforation Diameter | inches | 1.0 | 0.0 | | Perforation Diameter | m | 0.0254 | 0.0 | | Perforation Diameter | feet | 0.08333333 | 0.0 | | Perforation Diameter | cm | 2.54 | 0.0 | | Perforation Diameter | mm | 25.4 | 0.0 | | Perforation Diameter | km | 2.54e-05 | 0.0 | | Perforation Diameter | miles | 1.58e-05 | 0.0 | | Perforation Diameter | 64ths inch | 64.0 | 0.0 | | Perforation Efficiency | fraction | 1.0 | 0.0 | | Perforation Efficiency | percent | 100.0 | 0.0 | | Perforation Interval | feet | 1.0 | 0.0 | | Perforation Interval | m | 0.3048 | 0.0 | | Perforation Interval | miles | 0.000189393 | 0.0 | | Perforation Interval | km | 0.0003048 | 0.0 | | Perforation Interval | inches | 12.0 | 0.0 | | Perforation Interval | cm | 30.48 | 0.0 | | Perforation Interval | mm | 304.8 | 0.0 | | Perforation Interval | 64ths inch | 768.0 | 0.0 | | Perforation Length | inches | 1.0 | 0.0 | | Perforation Length | m | 0.0254 | 0.0 | | Perforation Length | feet | 0.08333333 | 0.0 | | Perforation Length | cm | 2.54 | 0.0 | | Perforation Length | mm | 25.4 | 0.0 | | Perforation Length | km | 2.54e-05 | 0.0 | | Perforation Length | miles | 1.58e-05 | 0.0 | | Perforation Length | 64ths inch | 64.0 | 0.0 | | Permeability | md | 1.0 | 0.0 | | Permeability | um2 | 0.0009869 | 0.0 | | Permeability | darcy | 0.001 | 0.0 | | Pipe Displacement | feet | 1.0 | 0.0 | | Pipe Displacement | m | 0.3048 | 0.0 | | Pipe Displacement | miles | 0.000189393 | 0.0 | | Pipe Displacement | km | 0.0003048 | 0.0 | | Pipe Displacement | inches | 12.0 | 0.0 | | Pipe Displacement | cm | 30.48 | 0.0 | | Pipe Displacement | mm | 304.8 | 0.0 | | Pipe Displacement | 64ths inch | 768.0 | 0.0 | | Pipe Length | feet | 1.0 | 0.0 | | Pipe Length | m | 0.3048 | 0.0 | | Pipe Length | miles | 0.000189393 | 0.0 | | Pipe Length | km | 0.0003048 | 0.0 | | Pipe Length | inches | 12.0 | 0.0 | | Pipe Length | cm | 30.48 | 0.0 | | Pipe Length | mm | 304.8 | 0.0 | | Pipe Length | 64ths inch | 768.0 | 0.0 | | Porosity | fraction | 1.0 | 0.0 | | Porosity | percent | 100.0 | 0.0 | | Power | hp | 1.0 | 0.0 | | Power | kW | 0.7457 | 0.0 | | Power Cost | US\$/hp | 1.0 | 0.0 | | Power Cost | US\$/kW | 0.745701 | 0.0 | | Pressure | psig | 1.0 | 0.0 | | Pressure | psia | 1.0 | 14.696 | | Pressure | BARa | 0.06894757 | 14.696 | | Pressure | kPa a | 6.894757 | 14.696 | | Pressure | kPa g | 6.894757 | 0.0 | | Pressure | BARg | 0.06894757 | 0.0 | | Pressure | Kg/cm2 a | 0.07031 | 14.696 | | Pressure | Kg/cm2 g | 0.07031 | 0.0 | | Pressure | atm a | 0.068046 | 14.696 | | Pressure Difference | psi | 1.0 | 0.0 | | Pressure Difference | bar | 0.06894757 | 0.0 | | Pressure Difference | kPa | 6.894757 | 0.0 | | Pressure Difference | Kg/cm2 | 0.07031 | 0.0 | | Pressure Difference | Pa | 6894.757 | 0.0 | | Pressure Gradient | psi/ft | 1.0 | 0.0 | | Pressure Gradient | bar/m | 0.2262059 | 0.0 | | Pressure Gradient | kPa/m | 22.62059 | 0.0 | | Pressure Gradient | Kg/cm2/m | 0.2306714 | 0.0 | | Pressure Gradient | Pa/m | 22620.59 | 0.0 | | Pressure Gradient | psi/m | 3.2808 | 0.0 | | Pressure Ratio | fraction | 1.0 | 0.0 | | Pressure Ratio | percent | 100.0 | 0.0 | | Productivity Index (Gas) | MMscf/day/psi | 1.0 | 0.0 | | Productivity Index (Gas) | Sm3/day/bar | 408462.5455 | 0.0 | | Productivity Index (Gas) | m3/(d.kPa) | 4084.6255 | 0.0 | | Productivity Index (Gas) | m3/d/bar | 408462.5455 | 0.0 | | Productivity Index (Oil) | STB/day/psi | 1.0 | 0.0 | | Productivity Index (Oil) | Sm3/day/bar | 2.305916 | 0.0 | | Productivity Index (Oil) | m3/d/kPa | 0.0231 | 0.0 | | Productivity Index (Oil) | m3/d/bar | 2.305916 | 0.0 | | Productivity Index (Oil) | Sm3/d/(kg/cm2) | 2.261233 | 0.0 | | Productivity Index Pseudo Pressure | (MMscf/day)/(psi2/cp) | 1.0 | 0.0 | | Productivity Index Pseudo Pressure | (Mscf/day)/(psi2/cp) | 1000.0 | 0.0 | | Productivity Index Pseudo Pressure | (Sm3/day)/(bar2/Pa.s) | 5926.666157597 | 0.0 | | Productivity Index Pseudo Pressure | (Sm3/day)/(kPa2/Pa.s) | 0.5926666157597 | 0.0 | | Proportion | percent | 1.0 | 0.0 | | Proportion | fraction | 0.01 | 0.0 | | Proportional Valve M | Mscf/day/psi | 1.0 | 0.0 | | Proportional Valve M | scf/day/psi | 1000.0 | 0.0 | | Proportional Valve M | Sm3/day/bar | 408.6292 | 0.0 | | Proportional Valve M | m3\[Vn]/hr/bar | 16.13943 | 0.0 | | Pseudo Pressure | psi2/cp | 1.0 | 0.0 | | Pseudo Pressure | bar2/(Pa.s) | 4.753767 | 0.0 | | Pseudo Pressure | kPa2/(Pa.s) | 4753.767 | 0.0 | | Pseudo Pressure | (kg/cm2)2/(Pa.s) | 4.943496 | 0.0 | | Pump Efficiency | percent | 1.0 | 0.0 | | Pump Efficiency | fraction | 0.01 | 0.0 | | Pump Volume | in3 | 1.0 | 0.0 | | Pump Volume | cc | 16.387 | 0.0 | | Pump Wear Factor | fraction | 1.0 | 0.0 | | Pump Wear Factor | percent | 100.0 | 0.0 | | Pump/Compressor Gas Rate | RMMcf/day | 1.0 | 0.0 | | Pump/Compressor Gas Rate | 1000Rm3/d | 28.32 | 0.0 | | Pump/Compressor Gas Rate | MRm3/d | 0.02832 | 0.0 | | Pump/Compressor Gas Rate | Rm3/h | 1180.0 | 0.0 | | Rate of Pressure Change | psi/min | 1.0 | 0.0 | | Rate of Pressure Change | bar/min | 0.06894757 | 0.0 | | Rate of Pressure Change | kPa/min | 6.894757 | 0.0 | | Rate of Pressure Change | Kg/cm2/min | 0.07031 | 0.0 | | Rate of Pressure Change | atm/min | 0.068046 | 0.0 | | Ratio | fraction | 1.0 | 0.0 | | Ratio | percent | 100.0 | 0.0 | | Relative Permeability | fraction | 1.0 | 0.0 | | Relative Permeability | percent | 100.0 | 0.0 | | Reservoir Rate | RB/day | 1.0 | 0.0 | | Reservoir Rate | m3/day | 0.159 | 0.0 | | Reservoir Rate | m3/hour | 0.006625 | 0.0 | | Reservoir Thickness | feet | 1.0 | 0.0 | | Reservoir Thickness | m | 0.3048 | 0.0 | | Reservoir Thickness | miles | 0.000189393 | 0.0 | | Reservoir Thickness | km | 0.0003048 | 0.0 | | Reservoir Thickness | inches | 12.0 | 0.0 | | Reservoir Thickness | cm | 30.48 | 0.0 | | Reservoir Thickness | mm | 304.8 | 0.0 | | Reservoir Thickness | 64ths inch | 768.0 | 0.0 | | Reservoir Volume | MMft3 | 1.0 | 0.0 | | Reservoir Volume | Mm3 | 0.0283168 | 0.0 | | Reservoir Volume | Bft3 | 0.001 | 0.0 | | Reservoir Volume | Gm3 | 2.83168e-05 | 0.0 | | Reservoir Volume | Mm3Vn | 0.0283168 | 0.0 | | Reservoir Volume | Gm3Vn | 2.83168e-05 | 0.0 | | Reservoir Volume | MMRB | 0.1781973 | 0.0 | | Reservoir Volume | BRB | 0.0001781973 | 0.0 | | Reservoir Volume | acre-ft | 22.95684 | 0.0 | | Restriction Size | inches | 1.0 | 0.0 | | Restriction Size | m | 0.0254 | 0.0 | | Restriction Size | feet | 0.08333333 | 0.0 | | Restriction Size | cm | 2.54 | 0.0 | | Restriction Size | mm | 25.4 | 0.0 | | Restriction Size | km | 2.54e-05 | 0.0 | | Restriction Size | miles | 1.58e-05 | 0.0 | | Restriction Size | 64ths inch | 64.0 | 0.0 | | Rock Density | g/cc | 1.0 | 0.0 | | Rock Density | kg/m3 | 1000.0 | 0.0 | | Rock Density | lb/ft3 | 62.42797 | 0.0 | | Rock Density | g/L | 1000.0 | 0.0 | | Rock Density | kg/L | 1.0 | 0.0 | | Roughness | inches | 1.0 | 0.0 | | Roughness | m | 0.0254 | 0.0 | | Roughness | feet | 0.08333333 | 0.0 | | Roughness | cm | 2.54 | 0.0 | | Roughness | mm | 25.4 | 0.0 | | Roughness | km | 2.54e-05 | 0.0 | | Roughness | miles | 1.58e-05 | 0.0 | | Roughness | 64ths inch | 64.0 | 0.0 | | Sand Prod Ratio (Gas) | lb/MMscf | 1.0 | 0.0 | | Sand Prod Ratio (Gas) | kg/1000Sm3 | 0.0160185 | 0.0 | | Sand Prod Ratio (Liq) | lb/STB | 1.0 | 0.0 | | Sand Prod Ratio (Liq) | kg/Sm3 | 2.85300785 | 0.0 | | Saturation | fraction | 1.0 | 0.0 | | Saturation | percent | 100.0 | 0.0 | | Shaliness | fraction | 1.0 | 0.0 | | Shaliness | percent | 100.0 | 0.0 | | Shot Density | 1/ft | 1.0 | 0.0 | | Shot Density | 1/m | 3.28084 | 0.0 | | Slot Density | 1/ft | 1.0 | 0.0 | | Slot Density | 1/m | 3.28084 | 0.0 | | Slug Frequency | 1/hour | 1.0 | 0.0 | | Slug Frequency | 1/min | 0.01666 | 0.0 | | Slug Frequency | 1/sec | 0.0002777 | 0.0 | | Slug Frequency | 1/day | 24.0 | 0.0 | | Slug Length | feet | 1.0 | 0.0 | | Slug Length | m | 0.3048 | 0.0 | | Slug Length | miles | 0.000189393 | 0.0 | | Slug Length | km | 0.0003048 | 0.0 | | Slug Length | inches | 12.0 | 0.0 | | Slug Length | cm | 30.48 | 0.0 | | Slug Length | mm | 304.8 | 0.0 | | Slug Length | 64ths inch | 768.0 | 0.0 | | Slug Rate | ft3/sec | 1.0 | 0.0 | | Slug Rate | m3/sec | 0.028316847 | 0.0 | | Slug Rate | ft3/min | 60.0 | 0.0 | | Slug Rate | m3/min | 1.699010796 | 0.0 | | Slug Rate | ft3/hr | 3600.0 | 0.0 | | Slug Rate | m3/hr | 101.9406477 | 0.0 | | Slug Rate | ft3/day | 86400.0 | 0.0 | | Slug Rate | m3/day | 2446.575546 | 0.0 | | Sonic Log | sec/MMft | 1.0 | 0.0 | | Sonic Log | sec/MMm | 3.28084 | 0.0 | | Specific Enthalpy | BTU/lbmol | 1.0 | 0.0 | | Specific Enthalpy | J/gmol | 2.326 | 0.0 | | Specific Enthalpy | kJ/kmol | 2.326 | 0.0 | | Specific Entropy | BTU/lbmol.R | 1.0 | 0.0 | | Specific Entropy | J/gmol.K | 4.1868 | 0.0 | | Specific Entropy | KJ/Kmol.K | 4.1868 | 0.0 | | Specific Heat Capacity | BTU/lb/F | 1.0 | 0.0 | | Specific Heat Capacity | KJ/Kg/K | 4.1868 | 0.0 | | Specific Heat Capacity | J/Kg/K | 4186.8 | 0.0 | | Specific Volume | ft3/lb | 1.0 | 0.0 | | Specific Volume | m3/Kg | 0.062427817 | 0.0 | | Specific Volume | cc/g | 62.427817 | 0.0 | | Speed of Sound | ft/sec | 1.0 | 0.0 | | Speed of Sound | m/sec | 0.3048 | 0.0 | | Speed of Sound | miles/hr | 0.6818181 | 0.0 | | Speed of Sound | km/hour | 1.09728 | 0.0 | | Stim Rate | bbl/min | 1.0 | 0.0 | | Stim Rate | m3/min | 0.1589873 | 0.0 | | Stim Rate | ft3/min | 5.615 | 0.0 | | Structural Unbalance | lb | 1.0 | 0.0 | | Structural Unbalance | Kg | 0.45359237 | 0.0 | | Sucker Rod Pump Speed | strokes/min | 1.0 | 0.0 | | Sucker Rod Stroke Length | in | 1.0 | 0.0 | | Sucker Rod Stroke Length | m | 0.0254 | 0.0 | | Sucker Rod Stroke Length | ft | 0.08333333 | 0.0 | | Sucker Rod Stroke Length | cm | 2.54 | 0.0 | | Sucker Rod Stroke Length | mm | 25.4 | 0.0 | | Sucker Rod Stroke Length | yd | 0.02777777 | 0.0 | | Sucker Rod Velocity | in/sec | 1.0 | 0.0 | | Sucker Rod Velocity | cm/sec | 2.54 | 0.0 | | Sucker Rod Velocity | ft/sec | 0.08333333 | 0.0 | | Sucker Rod Velocity | m/sec | 0.0254 | 0.0 | | Surface Area | ft2 | 1.0 | 0.0 | | Surface Area | m2 | 0.09290394 | 0.0 | | Swelling Coefficient | microstrain.ft3/scf | 1.0 | 0.0 | | Swelling Coefficient | microstrain.m3/Sm3 | 1.0 | 0.0 | | TWC/UCS | psig | 1.0 | 0.0 | | TWC/UCS | psia | 1.0 | 14.696 | | TWC/UCS | BARa | 0.06894757 | 14.696 | | TWC/UCS | kPa a | 6.894757 | 14.696 | | TWC/UCS | kPa g | 6.894757 | 0.0 | | TWC/UCS | BARg | 0.06894757 | 0.0 | | TWC/UCS | Kg/cm2 a | 0.07031 | 14.696 | | TWC/UCS | Kg/cm2 g | 0.07031 | 0.0 | | TWC/UCS | atm a | 0.068046 | 14.696 | | Temperature | deg F | 1.0 | 0.0 | | Temperature | deg C | 0.5555555 | -32.0 | | Temperature | deg R | 1.0 | 459.67 | | Temperature | deg K | 0.5555555 | 459.67 | | Temperature Difference | deg F | 1.0 | 0.0 | | Temperature Difference | deg C | 0.5555555 | 0.0 | | Temperature Gradient | DegF/100ft | 1.0 | 0.0 | | Temperature Gradient | DegC/m | 0.01822689 | 0.0 | | Temperature Gradient | DegC/100m | 1.822689 | 0.0 | | Thermal Conductivity | BTU/F/ft/day | 1.0 | 0.0 | | Thermal Conductivity | J/K/m/s | 0.0721136 | 0.0 | | Thermal Conductivity | KJ/K/m/s | 7.21e-05 | 0.0 | | Thermal Expansion | 1/deg F | 1.0 | 0.0 | | Thermal Expansion | 1/deg C | 1.8 | 0.0 | | Thermal Gradient | deg F/ft | 1.0 | 0.0 | | Thermal Gradient | deg C/ft | 0.55556 | 0.0 | | Thermal Gradient | deg F/m | 3.28084 | 0.0 | | Thermal Gradient | deg C/m | 1.82269 | 0.0 | | Thermal Power | BTU/d | 1.0 | 0.0 | | Thermal Power | W | 0.012211 | 0.0 | | Thermal Power | MMBTU/d | 1e-06 | 0.0 | | Thermal Power | MW | 1.221e-08 | 0.0 | | Thermal Power | BBTU/d | 1e-09 | 0.0 | | Thermal Power | GW | 1.2211e-11 | 0.0 | | Thermal Power | TBTU/d | 1e-12 | 0.0 | | Thermal Power | TW | 1.2211e-14 | 0.0 | | Thermal Power | kW | 1.2211e-05 | 0.0 | | Time | days | 1.0 | 0.0 | | Time | hours | 24.0 | 0.0 | | Time | minutes | 1440.0 | 0.0 | | Time | seconds | 86400.0 | 0.0 | | Time To Surface | seconds | 1.0 | 0.0 | | Time To Surface | minutes | 0.0166666 | 0.0 | | Time To Surface | hours | 0.0002777778 | 0.0 | | Time To Surface | days | 1.16e-05 | 0.0 | | Torque | lb.ft | 1.0 | 0.0 | | Torque | lb.ins | 12.0 | 0.0 | | Torque | N m | 1.355818 | 0.0 | | Torque | kgf m | 0.138255 | 0.0 | | Tubing Depth | feet | 1.0 | 0.0 | | Tubing Depth | m | 0.3048 | 0.0 | | Tubing Depth | miles | 0.000189393 | 0.0 | | Tubing Depth | km | 0.0003048 | 0.0 | | Tubing Depth | inches | 12.0 | 0.0 | | Tubing Depth | cm | 30.48 | 0.0 | | Tubing Depth | mm | 304.8 | 0.0 | | Tubing Depth | 64ths inch | 768.0 | 0.0 | | Turbine Flow | bbl/day | 1.0 | 0.0 | | Turbine Flow | gals/min | 0.02428633 | 0.0 | | Turbine Flow | m3/day | 0.159 | 0.0 | | Turbine Flow | m3/hour | 0.006625 | 0.0 | | Turbine Torque | lb.ins | 1.0 | 0.0 | | Turbine Torque | lb.ft | 0.0833333 | 0.0 | | Turbine Torque | N m | 0.1129848 | 0.0 | | Turbine Torque | kgf m | 0.011521246 | 0.0 | | Turbulence Coefficient | 1/ft | 1.0 | 0.0 | | Turbulence Coefficient | 1/m | 3.28084 | 0.0 | | Valve Size | 64ths inch | 1.0 | 0.0 | | Valve Size | inches | 0.015625 | 0.0 | | Valve Size | mm | 0.396875 | 0.0 | | Velocity | ft/sec | 1.0 | 0.0 | | Velocity | m/sec | 0.3048 | 0.0 | | Velocity | miles/hr | 0.6818181 | 0.0 | | Velocity | km/hour | 1.09728 | 0.0 | | Viscosity | centipoise | 1.0 | 0.0 | | Viscosity | Pa.s | 0.001 | 0.0 | | Viscosity | mPa.s | 1.0 | 0.0 | | Viscosity | uPa.s | 1000.0 | 0.0 | | Viscosity | poise | 0.01 | 0.0 | | Voltage | Volts | 1.0 | 0.0 | | Voltage | KVolts | 0.001 | 0.0 | | Voltage Drop | Volts/1000ft | 1.0 | 0.0 | | Voltage Drop | Volts/1000m | 3.28084 | 0.0 | | Voltage Drop | Volts/ft | 0.001 | 0.0 | | Voltage Drop | Volts/m | 0.00328084 | 0.0 | | Volume (scf) | scf | 1.0 | 0.0 | | Volume (scf) | Sm3 | 0.0283168 | 0.0 | | Volume Gallons US | Gal (US) | 1.0 | 0.0 | | Volume Gallons US | m3 | 0.0037854118 | 0.0 | | Volume Gallons US | ft3 | 0.133681 | 0.0 | | Volume Gallons US | bbl | 0.03174605 | 0.0 | | Water Cost | US\$/STB | 1.0 | 0.0 | | Water Cost | US\$/Sm3 | 6.289 | 0.0 | | Water Cost | US\$/m3 | 6.289 | 0.0 | | Water Cost | US\$/\[Vn]m3 | 6.289 | 0.0 | | Water Cut | percent | 1.0 | 0.0 | | Water Cut | fraction | 0.01 | 0.0 | | Water Gravity | ppm | 1.0 | 0.0 | | Water Gravity | percent NaCl | 0.0001 | 0.0 | | Water Gravity | Kg/Kg | 1e-06 | 0.0 | | Water Gravity | sp. gravity | 6.89e-07 | 1451378.0 | | Water Influx | MMSTB | 1.0 | 0.0 | | Water Influx | MSm3 | 0.159 | 0.0 | | Water Influx | BSTB | 0.001 | 0.0 | | Water Influx | BSm3 | 0.000159 | 0.0 | | Water Influx | MSTB | 1000.0 | 0.0 | | Water Influx | kSm3 | 159.0 | 0.0 | | Water Influx | Mm3 | 0.159 | 0.0 | | Water Influx | km3 | 159.0 | 0.0 | | Water Influx | GSm3 | 0.000159 | 0.0 | | Water Influx | Gm3 | 0.000159 | 0.0 | | Water Influx | STB | 1000000.0 | 0.0 | | Water Influx | Sm3 | 159000.0 | 0.0 | | Water Influx | m3 | 159000.0 | 0.0 | | Water Injection | MMSTB | 1.0 | 0.0 | | Water Injection | MSm3 | 0.159 | 0.0 | | Water Injection | BSTB | 0.001 | 0.0 | | Water Injection | BSm3 | 0.000159 | 0.0 | | Water Injection | MSTB | 1000.0 | 0.0 | | Water Injection | kSm3 | 159.0 | 0.0 | | Water Injection | Mm3 | 0.159 | 0.0 | | Water Injection | km3 | 159.0 | 0.0 | | Water Injection | GSm3 | 0.000159 | 0.0 | | Water Injection | Gm3 | 0.000159 | 0.0 | | Water Injection | STB | 1000000.0 | 0.0 | | Water Injection | Sm3 | 159000.0 | 0.0 | | Water Injection | m3 | 159000.0 | 0.0 | | Water Production | MMSTB | 1.0 | 0.0 | | Water Production | MSm3 | 0.159 | 0.0 | | Water Production | BSTB | 0.001 | 0.0 | | Water Production | BSm3 | 0.000159 | 0.0 | | Water Production | MSTB | 1000.0 | 0.0 | | Water Production | kSm3 | 159.0 | 0.0 | | Water Production | Mm3 | 0.159 | 0.0 | | Water Production | km3 | 159.0 | 0.0 | | Water Production | GSm3 | 0.000159 | 0.0 | | Water Production | Gm3 | 0.000159 | 0.0 | | Water Production | STB | 1000000.0 | 0.0 | | Water Production | Sm3 | 159000.0 | 0.0 | | Water Production | m3 | 159000.0 | 0.0 | | Water Salinity | ppm | 1.0 | 0.0 | | Water Salinity | percent NaCl | 0.0001 | 0.0 | | Water Salinity | Kg/Kg | 1e-06 | 0.0 | | Water Salinity | sp. gravity | 6.89e-07 | 1451378.0 | | Weight Cost | US\$/lbm | 1.0 | 0.0 | | Weight Cost | US\$/Kg | 2.204622621649 | 0.0 | | Weight Cost | US\$/s. ton | 2000.0 | 0.0 | | Weight Cost | US\$/l. ton | 2240.0 | 0.0 | | Weight Cost | US\$/tonne | 2204.622621849 | 0.0 | | Well kh | mD.ft | 1.0 | 0.0 | | Well kh | D.ft | 0.001 | 0.0 | | Well kh | mD.m | 0.3048 | 0.0 | | Well kh | D.m | 0.0003048 | 0.0 | | Wellbore Radius | feet | 1.0 | 0.0 | | Wellbore Radius | m | 0.3048 | 0.0 | | Wellbore Radius | miles | 0.000189393 | 0.0 | | Wellbore Radius | km | 0.0003048 | 0.0 | | Wellbore Radius | inches | 12.0 | 0.0 | | Wellbore Radius | cm | 30.48 | 0.0 | | Wellbore Radius | mm | 304.8 | 0.0 | | Wellbore Radius | 64ths inch | 768.0 | 0.0 | | Year | Year | 1.0 | 0.0 | | Young's Modulus | psi | 1.0 | 0.0 | | Young's Modulus | bar | 0.06894757 | 0.0 | | Young's Modulus | kPa | 6.894757 | 0.0 | | Young's Modulus | Kg/cm2 | 0.07031 | 0.0 | | Young's Modulus | Pa | 6894.757 | 0.0 | # Configure simulator connectors Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/index Learn how to set up and configure Windows-based simulator connectors to enable communication between simulation tools and Cognite Data Fusion (CDF). The connectors act as translators between your simulator's specific commands and CDF, facilitating bidirectional data flow while operating as a client making outbound https-connections only. ## Before you start **Infrastructure setup** * A Windows virtual or physical machine with storage space for model files. * Simulation software installed with valid licenses. * Windows administrator access for service configuration. **Authentication setup** * Registered service application in your Identity Provider (IdP). * Client secret for authentication. * Properly configured access groups in your IdP. * Dedicated data set in CDF for storing simulator data. * Appropriate [CDF group access capabilities](/cdf/access/guides/capabilities#simulator-connectors). **Network configuration** * Firewall allowing outbound traffic to `https://{cluster}.cognitedata.com`. * TLS certificate validation settings. * Proxy configuration (if applicable). ## Configuration settings To configure a simulator connector, you must edit the configuration file. The file is in [YAML](https://yaml.org) format, and you'll find configuration file examples listed under the `config` folder in the directory where the connector is installed. Don't base your configuration on the `config.example.yml` file. Instead, use `config.minimal.yml` as your base and copy the parts you need from `config.example.yml`. 1. Download simulator connectors from Data fusion > Integrate > **Extractors and connectors**. 2. Create `config.yml` in the connector's installation directory. Start with `config.minimal.yml` as a baseline. 3. Configure authentication values using your IdP credentials. 1. Create a dedicated Windows service/user account on the connector machine. 2. Set `read` and `write` access for the service account to the connector installation folder. 1. Configure the Windows service to use the local dedicated account. 2. Start the Windows service. 3. Monitor startup using Windows Event Viewer. 4. Verify the connector registration under **Integrations** in the **Simulators** tool in the CDF user interface or by listing *simulator integrations* through the Cognite API. * When the connector starts, it reads the configuration file, authenticates with CDF, registers itself as a simulator integration, and starts processing models and routines. * When you've configured the connector, it registers itself and the associated simulator in CDF, and you can create models and routines. For on-premises connectors, customers are responsible for maintaining connector uptime. Connectors are designed to operate as always-on services to ensure consistent compute availability for users. ## Using values from environment variables In the configuration file, values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. ## Configuration sections ### Connector This section contains information on how the current connector appears in **Simulators**. | Parameter | Description | Required | | - | - | - | | **connector.data-set-id** | Enter the data set ID to read the model files and simulator routines from. Result inputs and outputs will be written to this data set. | Yes | | **connector.name-prefix** | Enter the connector name that is registered on the API. | No | | **connector.add-machine-name-suffix** | When set to `true` (default), the connector name is composed of the prefix `name-prefix@vm-host-name` followed by the machine name. | No | | **connector.simulation-run-tolerance** | The connector will time out simulation runs that are older than this value (in seconds). | No | | **connector.fetch-runs-interval** | The connector will fetch simulation runs from CDF at this interval (in seconds). | No | | **connector.status-interval** | The connector will update its heartbeat in CDF at this interval (in seconds). | No | | **connector.scheduler-update-interval** | The connector will check if scheduled simulations should be triggered at this interval (in seconds) | No | | **connector.simulation-run-load-balancing-enabled** | When set to `true`, the connector participates in **[Simulation run load balancing](/cdf/integration/guides/simulators/simulator_api)** (*[preview](/cdf/product_feature_status)*), and CDF assigns queued runs across healthy connectors for higher throughput. The default value is `false`, which preserves earlier behavior: runs stay tied to your connector. | No | | **connector.simulation-run-poll-limit** | The maximum number of simulation runs the connector gets assigned at a time. When `connector.simulation-run-load-balancing-enabled` is `true` (*[preview](/cdf/product_feature_status)*), queued runs and runs assigned to this connector count together toward this limit. The default value is `10`. | No | | **connector.model-library.model-parsing-timeout** | The maximum time, in seconds, within which the connector must finish parsing a model revision when `connector.simulation-run-load-balancing-enabled` is `true`. If parsing isn't complete within this time, the connector marks the revision as `failure`. The default value is `3600` (1 hour). | No | ### Logger Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, `Verbose`, in order of decreasing importance. Each level covers the ones of higher importance. | Parameter | Description | | - | - | | **console** | Configuration for logging to the console. | | **console.level** | Set the minimum level of log events to write to the console. Set this parameter to enable console logging. | | **console.stderr-level** | Log events at this level or above are redirected to standard error. | | **file** | Configuration for logging to a rotating log file. | | **file.level** | Set the minimum level of log events to write to file. | | **file.path** | Enter the path to the files to be logged. For instance, if this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | **file.retention-limit** | Set the maximum number of log files that are kept in the log folder. | | **file.rolling-interval** | Enter a rolling interval for log files as `day` or `hour`. The default value is `day`. | | **remote** | Configuration for logging to CDF. | | **remote.enabled** | Insert boolean of whether to push logs to CDF or not. The default value is `true`. | | **remote.level** | Set the minimum level of log events to push to CDF. Valid values are `Error`, `Warning`, `Information` and `Debug`. | ### Cognite This is the configuration of the connection to CDF. | Parameter | Description | | - | - | | **host** | Insert the CDF service URL. The default value is `https://api.cognitedata.com`. | | **project** | Enter the CDF project. This is a required parameter. | | **idp-authentication** | The configuration for authenticating to CDF. | | **idp-authentication.authority** | The authority used with `tenant` to authenticate to Azure tenants. Use `token-url` if you're connecting using a non-azure IdP. The default value is `https://login.microsoftonline.com`. | | **idp-authentication.tenant** | Enter the Azure tenant used with `authority`. | | **idp-authentication.token-url** | Insert the URL used to obtain service tokens, used for non-azure IdPs. | | **idp-authentication.client-id** | Enter the service principal client ID. | | **idp-authentication.secret** | Enter the service principal client secret. | | **idp-authentication.resource** | Insert an optional resource parameter to pass along with token requests. | | **idp-authentication.scopes** | Enter a list of scopes to pass along with the request. This will typically need to contain `[host]/.default`. | | **idp-authentication.audience** | Insert an optional audience parameter to pass along with token requests. | | **idp-authentication.min-ttl** | Set the requested minimum time-to-live in seconds for the token. | | **idp-authentication.certificate** | Configuration for authenticating using a client certificate. | | **idp-authentication.certificate.authority-url** | Insert the certificate authority URL. | | **idp-authentication.certificate.path** | Enter the path to the `.pem` or `.pfx` certificate to be used for authentication. | | **idp-authentication.certificate.password** | Enter the certificate password. | | **max-data-points-upload-queue-size** | Set the maximum number of cached data points before they are uploaded to CDF. The default value is `1000000`. | | **cdf-retries** | Set the number of automatic retries on requests to CDF. | | **cdf-retries.timeout** | Set a timeout in milliseconds for each try. The default value is `80000`. | | **cdf-retries.max-retries** | Set the maximum number of retries. A value less than 0 retries forever. | | **cdf-retries.max-delay** | Set the maximum delay between each try in milliseconds. The base delay is calculated according to `125 * 2 ^ retry` milliseconds. If this is less than 0, there's no upper limit. The default value is `5000`. | | **cdf-chunking** | The configuration for chunking on requests to CDF. Note that increasing these may cause requests to fail, due to limits in the API. Read the API documentation before making these higher than their current value. | | **cdf-chunking.time-series** | Set the maximum number of time series per get/create time series request. | | **cdf-chunking.data-point-time-series** | Set the maximum number of time series per data point create request. | | **cdf-chunking.data-points** | Set the maximum number of data points per data point create request. | | **cdf-throttling** | Set how requests to CDF should be throttled. | | **cdf-throttling.time-series** | Set the maximum number of parallel requests per time series operation. The default value is `20`. | | **cdf-throttling.data-points** | Set the maximum number of parallel requests per datapoints operation. The default value is `10`. | | **sdk-logging** | The configuration for logging of requests from the SDK. | | **sdk-logging.disable** | Set this to `true` to disable logging of requests from the SDK. This is by default enabled. | | **sdk-logging.level** | Set the log level to log messages from the SDK. The default value is `debug`. | | **sdk-logging.format** | Set the log message format. The default value is `CDF ({Message}): {HttpMethod} {Url} - {Elapsed} ms` | | **nan-replacement** | Replacement for `NaN` values when writing to CDF. The default value is none, meaning these values are removed. | | **extraction-pipeline.external-id** | The configuration for associating this connector with an extraction pipeline. This is used for monitoring and remote configuration. | | **certificates** | The configuration for special handling of SSL certificates. This should not be considered as a permanent solution to certificate problems. | | **certificates.accept-all** | Accept all remote SSL certificates even if verification fails. Note: This introduces a risk of man-in-the-middle attacks. | | **certificates.allow-list** | List of certificate thumbprints to automatically accept. This is a smaller risk than accepting all certificates. | ### State store This section includes the configuration parameters for storing state in a local database. | Parameter | Description | | - | - | | **location** | Enter the path to the database file containing the state store. | | **database** | Select which type of database to use. Only `LiteDb` is currently supported. The default value is `None`. | ## Maintain connectors * Connectors might require significant disk space for caching model files stored in CDF * Connectors don't have auto-update features, so regular manual updates are necessary to get bug fixes and new features * The underlying simulation software may also require periodic updates * When troubleshooting problems, check the connector logs in the installation directory and Windows Event Logs # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/prosper/prosper_configuration Learn how to configure the PROSPER connector with YAML parameters for automation and OpenServer connection settings. ### Minimal YAML configuration file ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: ${COGNITE_PROJECT # The API host to connect to example: https://api.cognitedata.com host: ${COGNITE_HOST} # Configure the number of retry attempts for network requests to CDF. # The following configuration results in approximately 5 minutes of exponential backoff # before ultimately failing. cdf-retries: max-retries: 12 max-delay: 60000 # This is for Microsoft as IdP, to use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} connector: # Enter the data set ID to read the model files and simulator routines from. # Result inputs and outputs will be written to this data set. (required) data-set-id: ${COGNITE_PROSPER_DATASET} ``` ## Connector Several connector settings can be configured in the `connector` section of the configuration file. The only required parameter is `connector.data-set-id`. | Parameter | Description | | - | - | | **connector.data-set-id** | Enter the data set ID to read the simulation resources from. Result inputs and outputs will be written to this data set. | ## Automation This section includes the parameters for configuring the PETEX OpenServer connection. This is configured in the `automation` section of the configuration file. | Parameter | Description | | - | - | | **automation.terminate-process** | Set to terminate the PROSPER and OpenServer process at the end of usage (model revision validation / simulation run) | | **automation.license-hold-time** | Time in seconds to hold the license for after an action (model revision validation / simulation run). | | **automation.simulator-retries** | Set the simulator connection retries. | | **automation.simulator-retries.max-tries** | Maximum amount of times to retry a connection failure to the simulator. | | **automation.simulator-retries.max-delay** | Maximum delay before each retry attempt to the simulator. For example, "10s" , "1m". | | **automation.open-server-async-command-timeout** | Maximum number of seconds to wait before timing out a "DoCommand" operation. For example, "600s". | | **automation.simulator-executable-path** | Path to the PROSPER executable. When set, the connector manages the simulator process lifecycle directly. Use official PETEX binaries because the connector verifies the executable's digital signature before launching it. | | **automation.skip-undefined-numeric-outputs** | If `true`, numeric outputs that PROSPER returns as `NaN`, `Infinity`, or `-Infinity` are logged and omitted from the simulation results. The simulation run continues. If any value in an array output is `NaN`, `Infinity`, or `-Infinity`, the connector omits the whole array. Default value is `false`. | ## Recommended configuration for simulation stability For stable runs, let the connector manage the simulator process lifecycle. When you set `simulator-executable-path`, the connector starts and terminates the simulator processes, which reduces orphaned processes and licenses being held after runs complete. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} connector: automation: simulator-executable-path: "C:\\Program Files\\Petroleum Experts\\IPM 12\\prosper.exe" license-hold-time: 120 terminate-process: true ``` By default, the connector only obtains a license when needed. However, if the license hold time is set to a period more than 60s, the connector retains the license for the specified period after the last simulation run or model revision validation. This improves throughput for batched simulation runs by avoiding the overhead of repeatedly opening and closing the simulator. # System requirements Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/prosper/prosper_server_requirements Review hardware, software, and network requirements for installing the PROSPER simulator connector on physical or virtual machines. ## Memory The default configuration requires the server to have a minimum of **8 GB RAM** and an additional **256 GB HD** for logging and storage. You can adjust the configuration to reduce memory usage. ## CPU For the default values in the configuration, the server needs a minimum of **4-core CPUs**. The memory and CPU requirements also depend on the requirements provided by PROSPER. The machine should fulfill the basic requirements needed to run PROSPER simulators. ## Software * Windows Server 2016 or later. * A Windows account with admin rights to install the connector. * Access to the Windows machine, for instance through VPN. * .NET 8.0 runtime. * PETEX OpenServer license. * The PETEX simulators; PROSPER or GAP. The connector has been tested on PETEX IPM v11.5 - v13.5. ## Dedicated environment The PROSPER connector requires a **dedicated environment** where it's the only service interacting with OpenServer. No other processes, connectors, third-party applications, or custom integrations can use OpenServer in the same environment If you encounter issues while running the connector in an environment with other OpenServer processes, we may be unable to offer support. **Recommendation:** Install the PROSPER connector on a **dedicated virtual machine (VM)** or **physical server**. ## Network access If the VM is behind a firewall, allow access to the endpoint: `https://{cluster}.cognitedata.com`. For example, if the project is in the Europe 2 (Microsoft) cluster, then this endpoint is [https://westeurope-1.cognitedata.com](https://westeurope-1.cognitedata.com). # Setup Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/prosper/prosper_setup Learn how to set up and configure the PROSPER connector Windows service and connect to the PROSPER simulator. ## Before you start Check the [server requirements](/cdf/integration/guides/simulators/connectors/prosper/prosper_server_requirements) for the connector. Make sure the connector has [these capabilities](/cdf/access/guides/capabilities#simulator-connectors) in a Cognite Data Fusion (CDF) project. Create a configuration file according to the [configuration settings](/cdf/integration/guides/simulators/connectors/prosper/prosper_configuration) in the working directory of the folder where the connector is installed. The file must be in YAML format. ## Connect to PROSPER simulator The connector uses COM to connect to the PETEX OpenServer protocol, which establishes the connection with the PROSPER simulator. In case of connection errors, it can retry. The retry parameters can be defined in the configuration file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} automation: simulator-retries: max-tries: 12 max-delay: "60s" ``` This configuration enables retry attempts to the simulator for up to 5 minutes before failing. The retries follow an exponential backoff strategy, starting with shorter delays and gradually increasing to a maximum interval of one minute. | Retry attempt | Delay duration before attempt | | - | - | | 1 | ░░ (0.5s) | | 2 | ░░░░ (1s) | | 3 | ▓▓▓▓▓ (2s) | | 4 | ██████ (4s) | | 5 | ██████████ (8s) | | 6 | ██████████████ (16s) | | 7 | ██████████████████ (32s) | | 8 | ████████████████████████ (60s limit) | | 9 | ████████████████████████ (60s limit) | | 10 | ████████████████████████ (60s limit) | ## Run as a Windows service Download the *prosper-setup-\{VERSION}.exe* file via the download links available from Data fusion > Integrate > **Extractors** in CDF. Run the installer on a Windows operating system. A wizard guides you through the installation process. Select the destination folder where the connector will be installed. By default, this is **C:\Cognite\\**. Inside this folder, the **ProsperConnector** working directory is created. The connector expects a `config.yml` file as the configuration in the same directory. You'll find templates for the configuration file in the `config` folder. Open the **Services** app in Windows and find the **Cognite PROSPER Connector** service. Right-click the service and select **Start**. ## Uninstall and update the connector Run the same MSI installer to uninstall the connector. The installer deletes installed binaries, removes the service, and cleans the application registry. The installation is not deleted and still contains the configuration file, logs, and application data generated by the connector. If the connector is re-installed, it resumes the operation using the saved state. When a new version of the PROSPER connector is available, run the MSI installer for the new version to upgrade the connector to the new version. After the upgrade, you must manually restart the service. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/prosper/prosper_troubleshooting Find solutions to common PROSPER connector issues including authentication errors, license problems, and file read failures. ## The connector receives 401 errors from CDF [HTTP error 401](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401) means that the service didn't recognize the authentication provided by the client. If the Cognite API returns 401 errors to the connector, it means the connector successfully received an authentication token, but the token isn't valid for the project. This usually occurs because the `project` or `host` parameters in the `cognite` section are incorrect. ## The connector is not setting the data set when using remote configuration If you set up the connector to run with remote configuration, only the `host`, `project`, `idp-authentication`, and `extraction-pipeline` parameters in the `cognite` section are read from the file on the local machine. Other parameters in the `cognite` section, such as `data-set`, must be added to the configuration file in the extraction pipelines. ## Diagnose license-related issues The connector will fail to run properly if you don't have a PETEX OpenServer license. The following PowerShell commands can be used to verify if PETEX OpenServer and PROSPER licenses are available in the environment where the connector is running. ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Create a new OpenServer COM object $server = New-Object -ComObject "PX32.OpenServer.1" # Test if a PROSPER license is available $server.DoCommand("PROSPER.START()") # Close PROSPER $server.DoCommand("PROSPER.SHUTDOWN") # Release the COM object [System.Runtime.Interopservices.Marshal]::ReleaseComObject($server) | Out-Null [System.GC]::Collect() [System.GC]::WaitForPendingFinalizers() ``` ## File read issues in model extraction The error message `"File Read Failed (3160)"` typically occurs when a model file can't be opened within the installed version of IPM. This error is typically due to a version mismatch. If a model file was saved with a newer version of the tools, for example IPM 13.5, and opened using OpenServer in an older version, for example IPM 13.0. The version mismatch prevents the file from being read. Projects in IPM Suite aren't backward-compatible due to structure amendments and new features.. # Unit conversion Source: https://docs.cognite.com/cdf/integration/guides/simulators/connectors/prosper/prosper_units Learn how the PROSPER connector handles unit conversion with supported quantities, units, and conversion factors. The PROSPER connector converts units internally based upon conversion factors. This functionality is implemented in the connector since OpenServer, the protocol used to integrate Cognite Data Fusion (CDF) with the PROSPER simulator, does not currently provide the ability for arbitrary unit conversion. When you configure a simulation routine revision, you can specify units for reading values from the simulator and units for setting values from CDF. If you don't specify a unit, values are transferred without conversion. Unit conversion can only be used for the variables types `DOUBLE` and `DOUBLE_ARRAY`. The conversion logic follows this formula: vtarget = (vsource - s) × m Where: | Symbol | Description | | - | - | | vtarget | The resulting value in the target unit. | | vsource | The original value in the source unit. | | s | The conversion shift factor. | | m | The conversion multiplier factor. | The conversion process follows these steps: Verify that the specified quantity exists in the unit conversion database. Confirm that the desired unit exists for the selected quantity. Retrieve the unit associated with the OpenServer address using the `UNITNAME` extension. Verify its existence in the unit conversion database for the selected quantity. Convert the units for both scalar and array values. * If you specify an invalid quantity or unit in your simulator routine, the connector will fail the simulation run. * The table below may not be comprehensive for new simulator releases. Customers can request the addition of new units as needed. This table lists all supported quantities, units, and their associated conversion factors from the PROSPER simulator unit management system: | Quantity | Unit | Multiplier | Shift | | - | - | - | - | | Acceleration | ft/sec/sec | 1.0 | 0.0 | | Acceleration | m/sec/sec | 0.3048 | 0.0 | | Angle | degrees | 1.0 | 0.0 | | Angle | radians | 0.0174533 | 0.0 | | Anisotropy | fraction | 1.0 | 0.0 | | Anisotropy | percent | 100.0 | 0.0 | | Area Per Length | in2/ft | 1.0 | 0.0 | | Area Per Length | ft2/ft | 0.006944444444 | 0.0 | | Area Per Length | cm2/m | 21.16666667 | 0.0 | | Area Per Length | m2/m | 0.002116666667 | 0.0 | | Bean Size | inches | 1.0 | 0.0 | | Bean Size | m | 0.0254 | 0.0 | | Bean Size | feet | 0.08333333 | 0.0 | | Bean Size | cm | 2.54 | 0.0 | | Bean Size | mm | 25.4 | 0.0 | | Bean Size | km | 2.54e-05 | 0.0 | | Bean Size | miles | 1.58e-05 | 0.0 | | Bean Size | 64ths inch | 64.0 | 0.0 | | Blade Height | mm | 1.0 | 0.0 | | Blade Height | ins | 25.4 | 0.0 | | Blade Height | cm | 10.0 | 0.0 | | Blade Height | 64ths | 0.396875 | 0.0 | | C (Gas) | Mscf/day/psi2 | 1.0 | 0.0 | | C (Gas) | Sm3/day/bar2 | 28.17399 | 0.0 | | C (Gas) | m3/day/kPa2 | 28.173 | 0.0 | | C (Gas) | m3Vn/h/bar2 | 1.11277 | 0.0 | | C (Oil) | STB/day/psi2 | 1.0 | 0.0 | | C (Oil) | Sm3/day/bar2 | 0.1589873 | 0.0 | | C (Oil) | m3/day/kPa2 | 0.1589873 | 0.0 | | C (Oil) | m3/hour/bar2 | 0.006625 | 0.0 | | CGR / WGR | STB/MMscf | 1.0 | 0.0 | | CGR / WGR | Sm3/Sm3 | 5.64e-06 | 0.0 | | CGR / WGR | m3/m3 | 5.64e-06 | 0.0 | | CGR / WGR | m3/m3Vn | 5.95e-06 | 0.0 | | CGR / WGR | STB/Mscf | 0.001 | 0.0 | | CGR / WGR | Sm3/kSm3 | 0.0056433 | 0.0 | | CGR / WGR | Sm3/MSm3 | 5.6433 | 0.0 | | Casing Weight | lb/ft | 1.0 | 0.0 | | Casing Weight | Kg/m | 1.488164 | 0.0 | | Casing Yield | Kpsi | 1.0 | 0.0 | | Casing Yield | Kbar | 0.06894757 | 0.0 | | Casing Yield | Pa | 68947570.0 | 0.0 | | Casing Yield | Kg/cm2 | 70.31 | 0.0 | | Casing Yield | psi | 1000.0 | 0.0 | | Casing Yield | bar | 68.94757 | 0.0 | | Choke Area | in2 | 1.0 | 0.0 | | Choke Area | cm2 | 6.45161 | 0.0 | | Choke Area | ft2 | 0.00694444 | 0.0 | | Choke Area | m2 | 0.000645161 | 0.0 | | Compressibility | 1/psi | 1.0 | 0.0 | | Compressibility | 1/bar | 14.503774 | 0.0 | | Compressibility | 1/Kpa | 0.14503774 | 0.0 | | Compressibility | 1/(Kg/cm2) | 14.22273 | 0.0 | | Conductivity | BTU/hr/ft/F | 1.0 | 0.0 | | Conductivity | W/m/K | 1.730735 | 0.0 | | Critical Pressure | psig | 1.0 | 0.0 | | Critical Pressure | psia | 1.0 | 14.696 | | Critical Pressure | BARa | 0.06894757 | 14.696 | | Critical Pressure | kPa a | 6.894757 | 14.696 | | Critical Pressure | kPa g | 6.894757 | 0.0 | | Critical Pressure | BARg | 0.06894757 | 0.0 | | Critical Pressure | Kg/cm2 a | 0.07031 | 14.696 | | Critical Pressure | Kg/cm2 g | 0.07031 | 0.0 | | Critical Pressure | atm a | 0.068046 | 14.696 | | Critical Temperature | deg F | 1.0 | 0.0 | | Critical Temperature | deg C | 0.5555555 | -32.0 | | Critical Temperature | deg R | 1.0 | 459.67 | | Critical Temperature | deg K | 0.5555555 | 459.67 | | Critical Volume | ft3/lb.mole | 1.0 | 0.0 | | Critical Volume | l/kg.mole | 62.4297 | 0.0 | | Critical Volume | m3/kg.mole | 0.06243 | 0.0 | | Cumulative Flow | ft3 | 1.0 | 0.0 | | Cumulative Flow | m3 | 0.02832 | 0.0 | | Currency | MMUS\$ | 1.0 | 0.0 | | Currency | US \$ | 1000000.0 | 0.0 | | Current | amps | 1.0 | 0.0 | | Damaged Zone Thickness | inches | 1.0 | 0.0 | | Damaged Zone Thickness | m | 0.0254 | 0.0 | | Damaged Zone Thickness | feet | 0.08333333 | 0.0 | | Damaged Zone Thickness | cm | 2.54 | 0.0 | | Damaged Zone Thickness | mm | 25.4 | 0.0 | | Damaged Zone Thickness | km | 2.54e-05 | 0.0 | | Damaged Zone Thickness | miles | 1.58e-05 | 0.0 | | Damaged Zone Thickness | 64ths inch | 64.0 | 0.0 | | Darcy Coefficient (Gas) | psi2/(Mscf/day) | 1.0 | 0.0 | | Darcy Coefficient (Gas) | bar2/(Sm3/day) | 0.0001685 | 0.0 | | Darcy Coefficient (Gas) | kPa2/(m3/day) | 1.687 | 0.0 | | Darcy Coefficient (Gas) | bar2/(m3\[Vn]/h) | 0.004272 | 0.0 | | Darcy Coefficient (Gas) | Kg/cm2/(Sm3/d) | 0.0001718296 | 0.0 | | Darcy Coefficient (Gas) | bar2/(MSm3/day) | 168.5 | 0.0 | | Darcy Coefficient (Oil) | psi/(STB/day) | 1.0 | 0.0 | | Darcy Coefficient (Oil) | bar/(Sm3/day) | 0.433667 | 0.0 | | Darcy Coefficient (Oil) | kPa/(m3/day) | 43.366715 | 0.0 | | Darcy Coefficient (Oil) | bar/(m3\[Vn]/h) | 10.408012 | 0.0 | | Darcy Coefficient (Oil) | kg/cm2/(Sm3/day) | 0.442217 | 0.0 | | Darcy Coefficient (Pseudo) | psi2/cp/(Mscf/day) | 1.0 | 0.0 | | Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(Sm3/day) | 0.1685 | 0.0 | | Darcy Coefficient (Pseudo) | kPa2/(Pa.s)/(m3/day) | 1687.0 | 0.0 | | Darcy Coefficient (Pseudo) | bar2/(Pa.s)/m3Vn/h) | 4.272 | 0.0 | | Darcy Coefficient (Pseudo) | (Kg/cm2)2/(Pa.s)/(Sm3/d) | 0.175225 | 0.0 | | Debris Weight | g | 1.0 | 0.0 | | Debris Weight | Kg | 0.001 | 0.0 | | Density | lb/ft3 | 1.0 | 0.0 | | Density | Kg/m3 | 16.0185 | 0.0 | | Density | g/cc | 0.0160185 | 0.0 | | Density | g/L | 16.0185 | 0.0 | | Density (g/L) | g/L | 1.0 | 0.0 | | Density (g/L) | lb/ft3 | 0.06242797 | 0.0 | | Density (g/L) | g/cc | 0.001 | 0.0 | | Density (g/L) | kg/m3 | 1.0 | 0.0 | | Density (g/L) | kg/L | 0.001 | 0.0 | | Diameter | inches | 1.0 | 0.0 | | Diameter | m | 0.0254 | 0.0 | | Diameter | feet | 0.08333333 | 0.0 | | Diameter | cm | 2.54 | 0.0 | | Diameter | mm | 25.4 | 0.0 | | Diameter | km | 2.54e-05 | 0.0 | | Diameter | miles | 1.58e-05 | 0.0 | | Diameter | 64ths inch | 64.0 | 0.0 | | Drainage Area | acres | 1.0 | 0.0 | | Drainage Area | m2 | 4046.9 | 0.0 | | Drainage Area | hectares | 0.40469 | 0.0 | | Drainage Area | ft2 | 43560.0 | 0.0 | | EOS Gas Volume | scf | 1.0 | 0.0 | | EOS Gas Volume | Sm3 | 0.0283168 | 0.0 | | EOS Mass | lbm | 1.0 | 0.0 | | EOS Mass | Kg | 0.45359237 | 0.0 | | EOS Mass | s. ton | 0.0005 | 0.0 | | EOS Mass | l. ton | 0.0004464285714 | 0.0 | | EOS Mass | tonne | 0.00045359237 | 0.0 | | EOS Oil Volume | STB | 1.0 | 0.0 | | EOS Oil Volume | Sm3 | 0.158987 | 0.0 | | EOS Oil Volume | scf | 5.615 | 0.0 | | Elastic Constant | in/lb/ft | 1.0 | 0.0 | | Elastic Constant | cm/kg/m | 18.3715 | 0.0 | | Energy per unit mass | J/Kg | 1.0 | 0.0 | | Energy per unit mass | KJ/Kg | 0.001 | 0.0 | | Energy per unit mass | BTU/lb | 0.4299226 | 0.0 | | Enthalpy | BTU/lb | 1.0 | 0.0 | | Enthalpy | J/gm | 2.326 | 0.0 | | Enthalpy | KJ/kg | 2.326 | 0.0 | | Entropy | BTU/lb.R | 1.0 | 0.0 | | Entropy | J/g.K | 4.1868 | 0.0 | | Entropy | KJ/Kg.K | 4.1868 | 0.0 | | Erosion | 0.001 inches/year | 1.0 | 0.0 | | Erosion | mm/year | 0.0254 | 0.0 | | Explosive Weight | g | 1.0 | 0.0 | | Explosive Weight | Kg | 0.001 | 0.0 | | Flow Duration | hours | 1.0 | 0.0 | | Flow Duration | minutes | 60.0 | 0.0 | | Flow Duration | seconds | 360.0 | 0.0 | | Flow Duration | days | 0.0416667 | 0.0 | | Fluid Gravity | sp. gravity | 1.0 | 0.0 | | Fluid Loss Time | minutes | 1.0 | 0.0 | | Fluid Loss Time | seconds | 60.0 | 0.0 | | Fluid Loss Time | hours | 0.016666667 | 0.0 | | Fluid Loss Time | days | 0.00069444444 | 0.0 | | Force | lbf | 1.0 | 0.0 | | Force | N | 4.448221 | 0.0 | | Fraction | fraction | 1.0 | 0.0 | | Fraction | percent | 100.0 | 0.0 | | Fracture Half Length | feet | 1.0 | 0.0 | | Fracture Half Length | m | 0.3048 | 0.0 | | Fracture Half Length | miles | 0.000189393 | 0.0 | | Fracture Half Length | km | 0.0003048 | 0.0 | | Fracture Half Length | inches | 12.0 | 0.0 | | Fracture Half Length | cm | 30.48 | 0.0 | | Fracture Half Length | mm | 304.8 | 0.0 | | Fracture Half Length | 64ths inch | 768.0 | 0.0 | | Fracture Height | feet | 1.0 | 0.0 | | Fracture Height | m | 0.3048 | 0.0 | | Fracture Height | miles | 0.000189393 | 0.0 | | Fracture Height | km | 0.0003048 | 0.0 | | Fracture Height | inches | 12.0 | 0.0 | | Fracture Height | cm | 30.48 | 0.0 | | Fracture Height | mm | 304.8 | 0.0 | | Fracture Height | 64ths inch | 768.0 | 0.0 | | Frequency | Hertz | 1.0 | 0.0 | | Frequency | kHz | 0.001 | 0.0 | | GOR / GLR | scf/STB | 1.0 | 0.0 | | GOR / GLR | Sm3/Sm3 | 0.1772 | 0.0 | | GOR / GLR | m3/m3 | 0.1772 | 0.0 | | GOR / GLR | m3Vn/m3 | 0.168 | 0.0 | | GOR / GLR | Mscf/STB | 0.001 | 0.0 | | GOR / GLR | kSm3/Sm3 | 0.0001772 | 0.0 | | Gas Cost | US\$/MMscf | 1.0 | 0.0 | | Gas Cost | US\$/1000Sm3 | 0.0355 | 0.0 | | Gas Cost | US\$/1000m3 | 0.0355 | 0.0 | | Gas Cost | US\$/1000\[Vn]m3 | 0.0355 | 0.0 | | Gas FVF | ft3/scf | 1.0 | 0.0 | | Gas FVF | m3/Sm3 | 1.005071 | 0.0 | | Gas FVF | m3/m3 (st) | 1.005071 | 0.0 | | Gas FVF | m3/m3 Vn | 1.060295 | 0.0 | | Gas FVF | RB/mcf | 178.0944 | 0.0 | | Gas FVF | RB/scf | 0.1780944 | 0.0 | | Gas FVF | RB/Mscf | 178.0944 | 0.0 | | Gas Gravity | sp. gravity | 1.0 | 0.0 | | Gas Gravity | lb/ft3 | 0.0764 | 0.0 | | Gas Gravity | kg/m3 | 1.22381 | 0.0 | | Gas Gravity | g/cc | 0.00122381 | 0.0 | | Gas Rate | MMscf/day | 1.0 | 0.0 | | Gas Rate | 1000Sm3/d | 28.17399 | 0.0 | | Gas Rate | 1000m3/d | 28.17399 | 0.0 | | Gas Rate | m3\[Vn]/h | 1112.774 | 0.0 | | Gas Rate | Mscf/day | 1000.0 | 0.0 | | Gas Rate | scf/day | 1000000.0 | 0.0 | | Gas Rate | Sm3/day | 28173.99 | 0.0 | | Gas Rate per Unit Length | MMscf/day/ft | 1.0 | 0.0 | | Gas Rate per Unit Length | 1000Sm3/d/m | 92.43435 | 0.0 | | Grain Diameter | inches | 1.0 | 0.0 | | Grain Diameter | m | 0.0254 | 0.0 | | Grain Diameter | feet | 0.08333333 | 0.0 | | Grain Diameter | cm | 2.54 | 0.0 | | Grain Diameter | mm | 25.4 | 0.0 | | Grain Diameter | km | 2.54e-05 | 0.0 | | Grain Diameter | miles | 1.58e-05 | 0.0 | | Grain Diameter | 64ths inch | 64.0 | 0.0 | | Gravel Pack Length | inches | 1.0 | 0.0 | | Gravel Pack Length | m | 0.0254 | 0.0 | | Gravel Pack Length | feet | 0.08333333 | 0.0 | | Gravel Pack Length | cm | 2.54 | 0.0 | | Gravel Pack Length | mm | 25.4 | 0.0 | | Gravel Pack Length | km | 2.54e-05 | 0.0 | | Gravel Pack Length | miles | 1.58e-05 | 0.0 | | Gravel Pack Length | 64ths inch | 64.0 | 0.0 | | Heat Transfer Coefficient | BTU/h/ft2/F | 1.0 | 0.0 | | Heat Transfer Coefficient | W/m2/K | 5.678983 | 0.0 | | Heat Transfer Coefficient | kJ/m2/h/K | 20.44175 | 0.0 | | Heating Value | BTU/scf | 1.0 | 0.0 | | Heating Value | kJ/sm3 | 37.25894617 | 0.0 | | Height Difference | inches | 1.0 | 0.0 | | Height Difference | m | 0.0254 | 0.0 | | Height Difference | feet | 0.08333333 | 0.0 | | Height Difference | cm | 2.54 | 0.0 | | Height Difference | mm | 25.4 | 0.0 | | Height Difference | km | 2.54e-05 | 0.0 | | Height Difference | miles | 1.58e-05 | 0.0 | | Height Difference | 64ths inch | 64.0 | 0.0 | | Humidity | percent | 1.0 | 0.0 | | Humidity | fraction | 0.01 | 0.0 | | Injected Gas Rate | Mscf/day | 1.0 | 0.0 | | Injected Gas Rate | scf/day | 1000.0 | 0.0 | | Injected Gas Rate | Sm3/day | 28.17399 | 0.0 | | Injected Gas Rate | m3\[Vn]/h | 1.1127743 | 0.0 | | Interfacial Tension | dyne/cm | 1.0 | 0.0 | | Interfacial Tension | mN/m | 1.0 | 0.0 | | Interfacial Tension | N/m | 0.001 | 0.0 | | Inverse Liquid Rate | day/STB | 1.0 | 0.0 | | Inverse Liquid Rate | day/Sm3 | 6.28981057 | 0.0 | | Inverse Liquid Rate | day/m3 | 6.28981057 | 0.0 | | Inverse Liquid Rate | hour/m3 | 150.94339 | 0.0 | | Inverse Liquid Rate | day/scf | 0.17809439 | 0.0 | | Joules Thomson Coefficient | degrees F/psi | 1.0 | 0.0 | | Joules Thomson Coefficient | degrees K/bar | 7.80556 | 0.0 | | Length | m | 0.3048 | 0.0 | | Length | feet | 1.0 | 0.0 | | Length | inches | 12.0 | 0.0 | | Length | cm | 30.48 | 0.0 | | Length | mm | 304.8 | 0.0 | | Length | km | 0.0003048 | 0.0 | | Length | miles | 0.000189393 | 0.0 | | Length | 64ths inch | 768.0 | 0.0 | | Linear Density | lb/ft | 1.0 | 0.0 | | Linear Density | Kg/m | 1.488164 | 0.0 | | Liquid Dropout | percent | 1.0 | 0.0 | | Liquid Dropout | fraction | 0.01 | 0.0 | | Liquid Rate | STB/day | 1.0 | 0.0 | | Liquid Rate | Sm3/day | 0.1589873 | 0.0 | | Liquid Rate | m3/day | 0.1589873 | 0.0 | | Liquid Rate | m3/hour | 0.006625 | 0.0 | | Liquid Rate | scf/day | 5.616 | 0.0 | | Liquid Rate (Reservoir) | RB/day | 1.0 | 0.0 | | Liquid Rate (Reservoir) | m3/day | 0.159 | 0.0 | | Liquid Rate (Reservoir) | m3/hour | 0.006625 | 0.0 | | Liquid Rate Per HP | STB/day/hp | 1.0 | 0.0 | | Liquid Rate Per HP | Sm3/day/kw | 0.11855683 | 0.0 | | Liquid Rate Per HP | m3/day/kw | 0.11855683 | 0.0 | | Liquid Rate Per HP | m3/hour/kw | 0.004940263 | 0.0 | | Liquid Rate Per HP | scf/day/hp | 5.615 | 0.0 | | Liquid Rate per Unit Length | STB/day/ft | 1.0 | 0.0 | | Liquid Rate per Unit Length | Sm3/day/m | 0.5216119 | 0.0 | | Liquid Volume (STB) | STB | 1.0 | 0.0 | | Liquid Volume (STB) | Sm3 | 0.158987 | 0.0 | | Liquid Volume (STB) | scf | 5.615 | 0.0 | | Log Permeability | md | 1.0 | 0.0 | | Log Permeability | um2 | 0.0009869 | 0.0 | | Log Permeability | darcy | 0.001 | 0.0 | | Mass | lbm | 1.0 | 0.0 | | Mass | Kg | 0.45359237 | 0.0 | | Mass | s. ton | 0.0005 | 0.0 | | Mass | l. ton | 0.0004464285714 | 0.0 | | Mass | tonne | 0.00045359237 | 0.0 | | Mass Concentration | kg/kg | 1.0 | 0.0 | | Mass Concentration | ppm | 1000000.0 | 0.0 | | Mass Concentration | mg/g | 1000.0 | 0.0 | | Mass Flow Rate | lbm/day | 1.0 | 0.0 | | Mass Flow Rate | Kg/day | 0.45359237 | 0.0 | | Mass Flow Rate | lbm/sec | 1.16e-05 | 0.0 | | Mass Flow Rate | s. ton/day | 0.0005 | 0.0 | | Mass Flow Rate | l. ton/day | 0.0004464285714 | 0.0 | | Mass Flow Rate | tonne/day | 0.00045359237 | 0.0 | | Mass Flow Rate | Kg/sec | 5.25e-06 | 0.0 | | Mass Flow Rate | lbm/hr | 0.04166666667 | 0.0 | | Mass Flow Rate | Kg/hr | 0.01889968208 | 0.0 | | Molar Flow Rate | lb-moles/day | 1.0 | 0.0 | | Molar Flow Rate | kg-moles/day | 0.45359237 | 0.0 | | Molecular Weight | lb/lb.mole | 1.0 | 0.0 | | Molecular Weight | g/g.mole | 1.0 | 0.0 | | Molecular Weight | kg/kg.mole | 1.0 | 0.0 | | Motor Speed | rpm | 1.0 | 0.0 | | Motor Speed | rps | 0.0166667 | 0.0 | | Motor Speed | Hertz | 0.0166667 | 0.0 | | Mud Density | lb/US gal | 1.0 | 0.0 | | Mud Density | lb/ft3 | 7.4805225 | 0.0 | | Mud Density | Kg/m3 | 119.8264 | 0.0 | | Non-Darcy Coefficient (Gas) | psi2/(Mscf/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(Sm3/day)2 | 5.98e-06 | 0.0 | | Non-Darcy Coefficient (Gas) | kPa2/(m3/day)2 | 0.05589 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(m3\[Vn]/h)2 | 0.003839 | 0.0 | | Non-Darcy Coefficient (Gas) | (Kg/cm2)2/(Sm3/d)2 | 6.22e-06 | 0.0 | | Non-Darcy Coefficient (Gas) | bar2/(MSm3/day)2 | 5980000.0 | 0.0 | | Non-Darcy Coefficient (Oil) | psi/(STB/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Oil) | bar/(Sm3/day)2 | 2.727684 | 0.0 | | Non-Darcy Coefficient (Oil) | kPa/(m3/day)2 | 272.7684 | 0.0 | | Non-Darcy Coefficient (Oil) | bar/(m3\[Vn]/h)2 | 1571.15 | 0.0 | | Non-Darcy Coefficient (Oil) | Kg/cm2/(Sm3/day)2 | 2.781464 | 0.0 | | Non-Darcy Coefficient (Pseudo) | psi2/cp/(Mscf/day)2 | 1.0 | 0.0 | | Non-Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(Sm3/day)2 | 0.00598 | 0.0 | | Non-Darcy Coefficient (Pseudo) | kPa2/(Pa.s)/(m3/day)2 | 55.89 | 0.0 | | Non-Darcy Coefficient (Pseudo) | bar2/(Pa.s)/(m3Vn/h)2 | 3.839 | 0.0 | | Non-Darcy Coefficient (Pseudo) | (kg/cm2)2/(Pa.s)/(Sm3/d)2 | 0.006218669 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(Mscf/day) | 1.0 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(Sm3/d) | 0.0354937 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(m3/d) | 0.0354937 | 0.0 | | Non-Darcy Flow Factor (Gas) | 1/(m3\[Vn]/h) | 0.0008986551 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(STB/day) | 1.0 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(Sm3/d) | 6.2898106 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(m3/d) | 6.2898106 | 0.0 | | Non-Darcy Flow Factor (Oil) | 1/(m3/hr) | 150.9434 | 0.0 | | Oil Cost | US\$/STB | 1.0 | 0.0 | | Oil Cost | US\$/Sm3 | 6.289 | 0.0 | | Oil Cost | US\$/m3 | 6.289 | 0.0 | | Oil Cost | US\$/\[Vn]m3 | 6.289 | 0.0 | | Oil FVF | RB/STB | 1.0 | 0.0 | | Oil FVF | m3/Sm3 | 1.0 | 0.0 | | Oil FVF | m3/m3 | 1.0 | 0.0 | | Oil Gravity | API | 1.0 | 0.0 | | Oil Gravity | rel.density | 141.5 | 131.5 | | Oil Gravity | Kg/m3 | 141500.0 | 131.5 | | OneSubsea Pump Rate | bbl/d | 1.0 | 0.0 | | OneSubsea Pump Rate | Am3/h | 0.006625 | 0.0 | | OneSubsea Pump Speed | rpm | 1.0 | 0.0 | | OneSubsea Pump Speed | HZ | 0.0166667 | 0.0 | | OneSubsea Pump dP | psi | 1.0 | 0.0 | | OneSubsea Pump dP | bar | 0.06894757 | 0.0 | | OneSubsea Pump dP | kPa | 6.894757 | 0.0 | | OneSubsea Pump dP | Kg/cm2 | 0.07031 | 0.0 | | OneSubsea Pump dP | Pa | 6894.757 | 0.0 | | Pad Volume | bbl | 1.0 | 0.0 | | Pad Volume | m3 | 0.1589873 | 0.0 | | Pad Volume | ft3 | 5.615 | 0.0 | | Penetration | fraction | 1.0 | 0.0 | | Penetration | percent | 100.0 | 0.0 | | Percent | percent | 1.0 | 0.0 | | Percent | fraction | 0.01 | 0.0 | | Perforation Diameter | inches | 1.0 | 0.0 | | Perforation Diameter | m | 0.0254 | 0.0 | | Perforation Diameter | feet | 0.08333333 | 0.0 | | Perforation Diameter | cm | 2.54 | 0.0 | | Perforation Diameter | mm | 25.4 | 0.0 | | Perforation Diameter | km | 2.54e-05 | 0.0 | | Perforation Diameter | miles | 1.58e-05 | 0.0 | | Perforation Diameter | 64ths inch | 64.0 | 0.0 | | Perforation Efficiency | fraction | 1.0 | 0.0 | | Perforation Efficiency | percent | 100.0 | 0.0 | | Perforation Interval | feet | 1.0 | 0.0 | | Perforation Interval | m | 0.3048 | 0.0 | | Perforation Interval | miles | 0.000189393 | 0.0 | | Perforation Interval | km | 0.0003048 | 0.0 | | Perforation Interval | inches | 12.0 | 0.0 | | Perforation Interval | cm | 30.48 | 0.0 | | Perforation Interval | mm | 304.8 | 0.0 | | Perforation Interval | 64ths inch | 768.0 | 0.0 | | Perforation Length | inches | 1.0 | 0.0 | | Perforation Length | m | 0.0254 | 0.0 | | Perforation Length | feet | 0.08333333 | 0.0 | | Perforation Length | cm | 2.54 | 0.0 | | Perforation Length | mm | 25.4 | 0.0 | | Perforation Length | km | 2.54e-05 | 0.0 | | Perforation Length | miles | 1.58e-05 | 0.0 | | Perforation Length | 64ths inch | 64.0 | 0.0 | | Permeability | md | 1.0 | 0.0 | | Permeability | um2 | 0.0009869 | 0.0 | | Permeability | darcy | 0.001 | 0.0 | | Pipe Displacement | feet | 1.0 | 0.0 | | Pipe Displacement | m | 0.3048 | 0.0 | | Pipe Displacement | miles | 0.000189393 | 0.0 | | Pipe Displacement | km | 0.0003048 | 0.0 | | Pipe Displacement | inches | 12.0 | 0.0 | | Pipe Displacement | cm | 30.48 | 0.0 | | Pipe Displacement | mm | 304.8 | 0.0 | | Pipe Displacement | 64ths inch | 768.0 | 0.0 | | Pipe Length | feet | 1.0 | 0.0 | | Pipe Length | m | 0.3048 | 0.0 | | Pipe Length | miles | 0.000189393 | 0.0 | | Pipe Length | km | 0.0003048 | 0.0 | | Pipe Length | inches | 12.0 | 0.0 | | Pipe Length | cm | 30.48 | 0.0 | | Pipe Length | mm | 304.8 | 0.0 | | Pipe Length | 64ths inch | 768.0 | 0.0 | | Porosity | fraction | 1.0 | 0.0 | | Porosity | percent | 100.0 | 0.0 | | Power | hp | 1.0 | 0.0 | | Power | kW | 0.7457 | 0.0 | | Pressure | psig | 1.0 | 0.0 | | Pressure | psia | 1.0 | 14.696 | | Pressure | BARg | 0.06894757 | 0.0 | | Pressure | BARa | 0.06894757 | 14.696 | | Pressure | kPa g | 6.894757 | 0.0 | | Pressure | kPa a | 6.894757 | 14.696 | | Pressure | Kg/cm2 g | 0.07031 | 0.0 | | Pressure | Kg/cm2 a | 0.07031 | 14.696 | | Pressure | atm a | 0.068046 | 14.696 | | Pressure Difference | psi | 1.0 | 0.0 | | Pressure Difference | bar | 0.06894757 | 0.0 | | Pressure Difference | kPa | 6.894757 | 0.0 | | Pressure Difference | Kg/cm2 | 0.07031 | 0.0 | | Pressure Difference | Pa | 6894.757 | 0.0 | | Pressure Gradient | psi/ft | 1.0 | 0.0 | | Pressure Gradient | bar/m | 0.2262059 | 0.0 | | Pressure Gradient | kPa/m | 22.62059 | 0.0 | | Pressure Gradient | Kg/cm2/m | 0.2306714 | 0.0 | | Pressure Gradient | Pa/m | 22620.59 | 0.0 | | Pressure Gradient | psi/m | 3.2808 | 0.0 | | Pressure Ratio | fraction | 1.0 | 0.0 | | Pressure Ratio | percent | 100.0 | 0.0 | | Productivity Index (Gas) | MMscf/day/psi | 1.0 | 0.0 | | Productivity Index (Gas) | Sm3/day/bar | 408462.5455 | 0.0 | | Productivity Index (Gas) | m3/(d.kPa) | 4084.6255 | 0.0 | | Productivity Index (Gas) | m3/d/bar | 408462.5455 | 0.0 | | Productivity Index (Oil) | STB/day/psi | 1.0 | 0.0 | | Productivity Index (Oil) | Sm3/day/bar | 2.305916 | 0.0 | | Productivity Index (Oil) | m3/d/kPa | 0.0231 | 0.0 | | Productivity Index (Oil) | m3/d/bar | 2.305916 | 0.0 | | Productivity Index (Oil) | Sm3/d/(kg/cm2) | 2.261233 | 0.0 | | Proportional Valve M | Mscf/day/psi | 1.0 | 0.0 | | Proportional Valve M | scf/day/psi | 1000.0 | 0.0 | | Proportional Valve M | Sm3/day/bar | 408.6292 | 0.0 | | Proportional Valve M | m3\[Vn]/hr/bar | 16.13943 | 0.0 | | Pseudo Pressure | psi2/cp | 1.0 | 0.0 | | Pseudo Pressure | bar2/(Pa.s) | 4.753767 | 0.0 | | Pseudo Pressure | kPa2/(Pa.s) | 4753.767 | 0.0 | | Pseudo Pressure | (kg/cm2)2/(Pa.s) | 4.943496 | 0.0 | | Pump Efficiency | percent | 1.0 | 0.0 | | Pump Efficiency | fraction | 0.01 | 0.0 | | Pump Volume | in3 | 1.0 | 0.0 | | Pump Volume | cc | 16.387 | 0.0 | | Pump Wear Factor | fraction | 1.0 | 0.0 | | Pump Wear Factor | percent | 100.0 | 0.0 | | Pump/Compressor Gas Rate | RMMcf/day | 1.0 | 0.0 | | Pump/Compressor Gas Rate | 1000Rm3/d | 28.32 | 0.0 | | Pump/Compressor Gas Rate | MRm3/d | 0.02832 | 0.0 | | Pump/Compressor Gas Rate | Rm3/h | 1180.0 | 0.0 | | Rate of Pressure Change | psi/min | 1.0 | 0.0 | | Rate of Pressure Change | bar/min | 0.06894757 | 0.0 | | Rate of Pressure Change | kPa/min | 6.894757 | 0.0 | | Rate of Pressure Change | Kg/cm2/min | 0.07031 | 0.0 | | Rate of Pressure Change | atm/min | 0.068046 | 0.0 | | Relative Permeability | fraction | 1.0 | 0.0 | | Relative Permeability | percent | 100.0 | 0.0 | | Reservoir Rate | RB/day | 1.0 | 0.0 | | Reservoir Rate | m3/day | 0.159 | 0.0 | | Reservoir Rate | m3/hour | 0.006625 | 0.0 | | Reservoir Thickness | feet | 1.0 | 0.0 | | Reservoir Thickness | m | 0.3048 | 0.0 | | Reservoir Thickness | miles | 0.000189393 | 0.0 | | Reservoir Thickness | km | 0.0003048 | 0.0 | | Reservoir Thickness | inches | 12.0 | 0.0 | | Reservoir Thickness | cm | 30.48 | 0.0 | | Reservoir Thickness | mm | 304.8 | 0.0 | | Reservoir Thickness | 64ths inch | 768.0 | 0.0 | | Resistance/Length | ohm/ft | 1.0 | 0.0 | | Resistance/Length | ohm/m | 3.28084 | 0.0 | | Restriction Size | inches | 1.0 | 0.0 | | Restriction Size | m | 0.0254 | 0.0 | | Restriction Size | feet | 0.08333333 | 0.0 | | Restriction Size | cm | 2.54 | 0.0 | | Restriction Size | mm | 25.4 | 0.0 | | Restriction Size | km | 2.54e-05 | 0.0 | | Restriction Size | miles | 1.58e-05 | 0.0 | | Restriction Size | 64ths inch | 64.0 | 0.0 | | Rock Density | g/cc | 1.0 | 0.0 | | Rock Density | kg/m3 | 1000.0 | 0.0 | | Rock Density | lb/ft3 | 62.42797 | 0.0 | | Rock Density | g/L | 1000.0 | 0.0 | | Rock Density | kg/L | 1.0 | 0.0 | | Roughness | inches | 1.0 | 0.0 | | Roughness | m | 0.0254 | 0.0 | | Roughness | feet | 0.08333333 | 0.0 | | Roughness | cm | 2.54 | 0.0 | | Roughness | mm | 25.4 | 0.0 | | Roughness | km | 2.54e-05 | 0.0 | | Roughness | miles | 1.58e-05 | 0.0 | | Roughness | 64ths inch | 64.0 | 0.0 | | Sand Prod Ratio (Gas) | lb/MMscf | 1.0 | 0.0 | | Sand Prod Ratio (Gas) | kg/1000Sm3 | 0.0160185 | 0.0 | | Sand Prod Ratio (Liq) | lb/STB | 1.0 | 0.0 | | Sand Prod Ratio (Liq) | kg/Sm3 | 2.85300785 | 0.0 | | Saturation | fraction | 1.0 | 0.0 | | Saturation | percent | 100.0 | 0.0 | | Shaliness | fraction | 1.0 | 0.0 | | Shaliness | percent | 100.0 | 0.0 | | Shot Density | 1/ft | 1.0 | 0.0 | | Shot Density | 1/m | 3.28084 | 0.0 | | Slot Density | 1/ft | 1.0 | 0.0 | | Slot Density | 1/m | 3.28084 | 0.0 | | Slug Frequency | 1/hour | 1.0 | 0.0 | | Slug Frequency | 1/min | 0.01666 | 0.0 | | Slug Frequency | 1/sec | 0.0002777 | 0.0 | | Slug Frequency | 1/day | 24.0 | 0.0 | | Slug Length | feet | 1.0 | 0.0 | | Slug Length | m | 0.3048 | 0.0 | | Slug Length | miles | 0.000189393 | 0.0 | | Slug Length | km | 0.0003048 | 0.0 | | Slug Length | inches | 12.0 | 0.0 | | Slug Length | cm | 30.48 | 0.0 | | Slug Length | mm | 304.8 | 0.0 | | Slug Length | 64ths inch | 768.0 | 0.0 | | Slug Rate | ft3/sec | 1.0 | 0.0 | | Slug Rate | m3/sec | 0.028316847 | 0.0 | | Slug Rate | ft3/min | 60.0 | 0.0 | | Slug Rate | m3/min | 1.699010796 | 0.0 | | Slug Rate | ft3/hr | 3600.0 | 0.0 | | Slug Rate | m3/hr | 101.9406477 | 0.0 | | Slug Rate | ft3/day | 86400.0 | 0.0 | | Slug Rate | m3/day | 2446.575546 | 0.0 | | Sonic Log | sec/MMft | 1.0 | 0.0 | | Sonic Log | sec/MMm | 3.28084 | 0.0 | | Specific Enthalpy | BTU/lbmol | 1.0 | 0.0 | | Specific Enthalpy | J/gmol | 2.326 | 0.0 | | Specific Enthalpy | kJ/kmol | 2.326 | 0.0 | | Specific Entropy | BTU/lbmol.R | 1.0 | 0.0 | | Specific Entropy | J/gmol.K | 4.1868 | 0.0 | | Specific Entropy | KJ/Kmol.K | 4.1868 | 0.0 | | Specific Heat Capacity | BTU/lb/F | 1.0 | 0.0 | | Specific Heat Capacity | KJ/Kg/K | 4.1868 | 0.0 | | Specific Heat Capacity | J/Kg/K | 4186.8 | 0.0 | | Specific Volume | ft3/lb | 1.0 | 0.0 | | Specific Volume | m3/Kg | 0.062427817 | 0.0 | | Specific Volume | cc/g | 62.427817 | 0.0 | | Speed of Sound | ft/sec | 1.0 | 0.0 | | Speed of Sound | m/sec | 0.3048 | 0.0 | | Speed of Sound | miles/hr | 0.6818181 | 0.0 | | Speed of Sound | km/hour | 1.09728 | 0.0 | | Stim Rate | bbl/min | 1.0 | 0.0 | | Stim Rate | m3/min | 0.1589873 | 0.0 | | Stim Rate | ft3/min | 5.615 | 0.0 | | Structural Unbalance | lb | 1.0 | 0.0 | | Structural Unbalance | Kg | 0.45359237 | 0.0 | | Sucker Rod Pump Speed | strokes/min | 1.0 | 0.0 | | Sucker Rod Stroke Length | in | 1.0 | 0.0 | | Sucker Rod Stroke Length | m | 0.0254 | 0.0 | | Sucker Rod Stroke Length | ft | 0.08333333 | 0.0 | | Sucker Rod Stroke Length | cm | 2.54 | 0.0 | | Sucker Rod Stroke Length | mm | 25.4 | 0.0 | | Sucker Rod Stroke Length | yd | 0.02777777 | 0.0 | | Sucker Rod Velocity | in/sec | 1.0 | 0.0 | | Sucker Rod Velocity | cm/sec | 2.54 | 0.0 | | Sucker Rod Velocity | ft/sec | 0.08333333 | 0.0 | | Sucker Rod Velocity | m/sec | 0.0254 | 0.0 | | TWC/UCS | psig | 1.0 | 0.0 | | TWC/UCS | psia | 1.0 | 14.696 | | TWC/UCS | BARa | 0.06894757 | 14.696 | | TWC/UCS | kPa a | 6.894757 | 14.696 | | TWC/UCS | kPa g | 6.894757 | 0.0 | | TWC/UCS | BARg | 0.06894757 | 0.0 | | TWC/UCS | Kg/cm2 a | 0.07031 | 14.696 | | TWC/UCS | Kg/cm2 g | 0.07031 | 0.0 | | TWC/UCS | atm a | 0.068046 | 14.696 | | Temperature | deg F | 1.0 | 0.0 | | Temperature | deg C | 0.5555555 | -32.0 | | Temperature | deg R | 1.0 | 459.67 | | Temperature | deg K | 0.5555555 | 459.67 | | Temperature Difference | deg F | 1.0 | 0.0 | | Temperature Difference | deg C | 0.5555555 | 0.0 | | Thermal Conductivity | BTU/F/ft/day | 1.0 | 0.0 | | Thermal Conductivity | J/K/m/s | 0.0721136 | 0.0 | | Thermal Conductivity | KJ/K/m/s | 7.21e-05 | 0.0 | | Thermal Expansion | 1/deg F | 1.0 | 0.0 | | Thermal Expansion | 1/deg C | 1.8 | 0.0 | | Thermal Gradient | deg F/ft | 1.0 | 0.0 | | Thermal Gradient | deg C/ft | 0.55556 | 0.0 | | Thermal Gradient | deg F/m | 3.28084 | 0.0 | | Thermal Gradient | deg C/m | 1.82269 | 0.0 | | Time | days | 1.0 | 0.0 | | Time | hours | 24.0 | 0.0 | | Time | minutes | 1440.0 | 0.0 | | Time | seconds | 86400.0 | 0.0 | | Time To Surface | seconds | 1.0 | 0.0 | | Time To Surface | minutes | 0.0166666 | 0.0 | | Time To Surface | hours | 0.0002777778 | 0.0 | | Time To Surface | days | 1.16e-05 | 0.0 | | Torque | lb.ft | 1.0 | 0.0 | | Torque | lb.ins | 12.0 | 0.0 | | Torque | N m | 1.355818 | 0.0 | | Torque | kgf m | 0.138255 | 0.0 | | Tubing Depth | feet | 1.0 | 0.0 | | Tubing Depth | m | 0.3048 | 0.0 | | Tubing Depth | miles | 0.000189393 | 0.0 | | Tubing Depth | km | 0.0003048 | 0.0 | | Tubing Depth | inches | 12.0 | 0.0 | | Tubing Depth | cm | 30.48 | 0.0 | | Tubing Depth | mm | 304.8 | 0.0 | | Tubing Depth | 64ths inch | 768.0 | 0.0 | | Turbine Flow | bbl/day | 1.0 | 0.0 | | Turbine Flow | gals/min | 0.02428633 | 0.0 | | Turbine Flow | m3/day | 0.159 | 0.0 | | Turbine Flow | m3/hour | 0.006625 | 0.0 | | Turbine Torque | lb.ins | 1.0 | 0.0 | | Turbine Torque | lb.ft | 0.0833333 | 0.0 | | Turbine Torque | N m | 0.1129848 | 0.0 | | Turbine Torque | kgf m | 0.011521246 | 0.0 | | Turbulence Coefficient | 1/ft | 1.0 | 0.0 | | Turbulence Coefficient | 1/m | 3.28084 | 0.0 | | Valve Size | 64ths inch | 1.0 | 0.0 | | Valve Size | inches | 0.015625 | 0.0 | | Valve Size | mm | 0.396875 | 0.0 | | Velocity | ft/sec | 1.0 | 0.0 | | Velocity | m/sec | 0.3048 | 0.0 | | Velocity | miles/hr | 0.6818181 | 0.0 | | Velocity | km/hour | 1.09728 | 0.0 | | Viscosity | centipoise | 1.0 | 0.0 | | Viscosity | Pa.s | 0.001 | 0.0 | | Viscosity | mPa.s | 1.0 | 0.0 | | Viscosity | uPa.s | 1000.0 | 0.0 | | Viscosity | poise | 0.01 | 0.0 | | Voltage | Volts | 1.0 | 0.0 | | Voltage | KVolts | 0.001 | 0.0 | | Voltage Drop | Volts/1000ft | 1.0 | 0.0 | | Voltage Drop | Volts/1000m | 3.28084 | 0.0 | | Voltage Drop | Volts/ft | 0.001 | 0.0 | | Voltage Drop | Volts/m | 0.00328084 | 0.0 | | Volume (scf) | scf | 1.0 | 0.0 | | Volume (scf) | Sm3 | 0.0283168 | 0.0 | | Volume Gallons US | Gal (US) | 1.0 | 0.0 | | Volume Gallons US | m3 | 0.0037854118 | 0.0 | | Volume Gallons US | ft3 | 0.133681 | 0.0 | | Volume Gallons US | bbl | 0.03174605 | 0.0 | | Water Cost | US\$/STB | 1.0 | 0.0 | | Water Cost | US\$/Sm3 | 6.289 | 0.0 | | Water Cost | US\$/m3 | 6.289 | 0.0 | | Water Cost | US\$/\[Vn]m3 | 6.289 | 0.0 | | Water Cut | percent | 1.0 | 0.0 | | Water Cut | fraction | 0.01 | 0.0 | | Water Salinity | ppm | 1.0 | 0.0 | | Water Salinity | percent NaCl | 0.0001 | 0.0 | | Water Salinity | Kg/Kg | 1e-06 | 0.0 | | Water Salinity | sp. gravity | 6.89e-07 | 1451378.0 | | Well kh | mD.ft | 1.0 | 0.0 | | Well kh | D.ft | 0.001 | 0.0 | | Well kh | mD.m | 0.3048 | 0.0 | | Well kh | D.m | 0.0003048 | 0.0 | | Wellbore Radius | feet | 1.0 | 0.0 | | Wellbore Radius | m | 0.3048 | 0.0 | | Wellbore Radius | miles | 0.000189393 | 0.0 | | Wellbore Radius | km | 0.0003048 | 0.0 | | Wellbore Radius | inches | 12.0 | 0.0 | | Wellbore Radius | cm | 30.48 | 0.0 | | Wellbore Radius | mm | 304.8 | 0.0 | | Wellbore Radius | 64ths inch | 768.0 | 0.0 | | Year | Year | 1.0 | 0.0 | | Young's Modulus | psi | 1.0 | 0.0 | | Young's Modulus | bar | 0.06894757 | 0.0 | | Young's Modulus | kPa | 6.894757 | 0.0 | | Young's Modulus | Kg/cm2 | 0.07031 | 0.0 | | Young's Modulus | Pa | 6894.757 | 0.0 | # About simulator integrations Source: https://docs.cognite.com/cdf/integration/guides/simulators/index Set up simulator integrations to connect first-principle simulation tools and CDF for real-time optimization, predictive maintenance, and digital twin development. Simulation integrations can be used for real-time asset performance optimization, predictive maintenance, what-if scenario analysis, and digital twin development. * Configure [simulator connectors](/cdf/integration/guides/simulators/connectors). Connectors are Windows services that run on the same host machine as external simulation tools. The connectors integrate external simulation tools with CDF by reading simulation models and simulator routines and use these to run simulations. The connectors use the information in the simulator routines to determine what data to read, which objects to interact with in the simulator, and the commands to run. * Create simulator models that represent an asset or process within a simulator. The simulator models act as the root of model revisions, routines, runs, and results. You can also upload the model files using the [Cognite Python SDK](https://cognite-sdk-python.readthedocs-hosted.com/en/latest). * Configure [simulator routines](/cdf/integration/guides/simulators/simulator_routines) to run simulations. A routine includes scheduling information, data preprocessing, data input and output mapping, and the simulator's execution steps. You can create multiple simulator routines for each simulation model to adapt to different simulation scenarios. * Manage and monitor simulations from Simulators under Data fusion > Build solutions in CDF. Create, edit, and [trigger](/cdf/integration/guides/simulators/simulator_trigger) simulation routines, monitor simulation runs, and explore simulation results. * With **[Simulation run load balancing](/cdf/integration/guides/simulators/simulator_api)** (*[preview](/cdf/product_feature_status)*), CDF can assign queued runs across healthy connectors for higher throughput. * You can use the [simulators API](/api-reference/concepts/20230101/simulators) to programmatically interact with [simulator resources in CDF](/cdf/integration/guides/simulators/simulator_api). The Simulators API enables all the state management and communication between CDF and the connectors. Simulator integration architecture diagram showing connections between CDF, connectors, and simulation tools # Simulators API Source: https://docs.cognite.com/cdf/integration/guides/simulators/simulator_api Learn about the Simulators API resources for managing simulation operations, from model definition to results analysis in Cognite Data Fusion (CDF). The Simulators API uses several Cognite API resources to enable comprehensive simulation operations. These resources work together to provide a complete system for managing simulation operations, from model definition to results analysis. [Simulators](/api-reference/concepts/20230101/simulators) contains the definitions necessary for CDF to interact with a given simulator. This resource serves as a central contract for APIs, user interfaces, and connectors to use the same definitions when dealing with a specific simulator. Each simulator is uniquely identified and can be associated with various file extension types, model types, step fields, and unit quantities. Simulators are essential for managing data flows between CDF and external simulation applications, ensuring consistency and reliability in data handling. [Simulator Integrations](/api-reference/concepts/20230101/simulator-integrations) represents a [simulator connector](/cdf/integration/guides/simulators/connectors) in CDF. This resource provides information about the configured connectors for a given simulator, such as status, data set, name, license status, connector version, simulator version. While the **simulator connector** is the active service performing the actual work, the **simulator integration** serves as its digital identity in CDF, providing real-time status monitoring through heartbeat information and managing relationships with other resources. This resource is essential for monitoring and managing the interactions between CDF and external simulation applications, ensuring proper data flow and integration. [Simulator Models](/api-reference/concepts/20230101/simulator-models) represent an asset modeled in a simulator. For instance, an asset can be a pump, a well, a processing facility, or a refinery. The simulator model is the root of its associated revisions, routines, runs, and results. The data set assigned to a model is inherited by its children. Deleting a model also deletes all its children, thereby maintaining the integrity and hierarchy of the simulation data. [Simulator Model Revisions](/api-reference/concepts/20230101/simulator-models) track changes and updates to simulator models to make sure modifications are traceable and to understand the evolution of a given model. Model revisions are immutable once created, providing reliable version control and history tracking. When a revision is created, a connector attempts to parse the model file, recording the success or failure of this operation. The parsing process can generate additional [Simulator Model Revision Data](/api-reference/concepts/20230101/simulator-models), which contains extracted information from the simulator file. This data resource includes structured information about the simulator model file. The type and extent of information extracted varies by simulator type and connector implementation, enabling the extraction of valuable metadata about the model's internal structure and configuration. ## Model revision limit Each simulator model can have up to 200 revisions. On `POST /simulators/models/revisions`, set `deleteOldest` to `true` on the request to create a new revision and delete the oldest revisions afterward, keeping only the newest 200. The parameter defaults to `false`. There is no connector configuration setting, and Simulators in CDF does not set this field. Rolling delete requires the `simulators:delete` capability on the relevant data set scope. See [Simulators users](/cdf/access/guides/capabilities#simulators-users). Without delete access, CDF returns HTTP 403 Forbidden. If `deleteOldest` is `false` or omitted and the model already has 200 revisions, CDF returns HTTP 422 with this message: `Resource limit exceeded for simulator model revision. Max 200 per model.` When rolling delete removes a model revision, CDF also deletes all [simulation runs](/api-reference/concepts/20230101/simulation-runs) that reference that revision, not only scheduled runs. Example request body: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "externalId": "pump-model-rev-201", "modelExternalId": "pump-model", "fileId": 1234567890123456 } ], "deleteOldest": true } ``` [Simulator Routines](/api-reference/concepts/20230101/simulator-routines) defines instructions for interactions with a simulator model. A simulator routine includes comprehensive specifications for how simulations should be executed, including inputs (values set into the simulator model), commands (actions to be performed by the simulator), and outputs (values read from the simulator model). Simulator routines can have multiple revisions, enabling users to track changes and evolve the routine over time. Each model can have multiple routines, each performing different objectives such as calculating optimal operation setpoints, forecasting production, benchmarking asset performance, and more. [Simulator Routine Revisions](/api-reference/concepts/20230101/simulator-routines) enable version control of [routine configurations](/cdf/integration/guides/simulators/simulator_routines), to track changes and evolve the routine over time. A routine revision consists of a configuration section and a script section. The configuration section defines settings for automated execution, data sampling parameters, and input/output specifications. The script section contains ordered groups and steps that define the execution sequence, with step types and arguments based on the simulator's capabilities. By default, a **simulator routine** references a **simulator integration**, so schedules and runs stay on that connector. Connectors use a queue system to manage simulation runs triggered by schedules and API calls, with configurable timeout settings to handle scenarios where incoming load exceeds processing capacity. You can distribute the load by creating multiple connectors and assigning routines across them for a manageable workload per connector. When you enable **[Simulation run load balancing](/cdf/integration/guides/simulators/connectors#connector)** (*[preview](/cdf/product_feature_status)*) for your CDF project, runs wait in a shared **FIFO** queue (first in, first out). **CDF assigns** each run to one healthy registered connector (**exactly-once execution**: one run, one connector). For higher throughput, install, configure, and start additional connector services on the machines where your simulators run. CDF does not create connector instances or run connector services in the cloud for you. See [Run simulations](/cdf/integration/guides/simulators/simulator_trigger) for how run status reflects assignment and execution. **Connectors** poll CDF for assigned runs on a configurable interval (`connector.fetch-runs-interval`), apply timeouts to stale runs, and emit logs when incoming load exceeds capacity. Every time a simulation routine executes, a [simulation run](/api-reference/concepts/20230101/simulation-runs) object is created to make sure each routine execution is documented and traceable. The simulation run tracks execution status, progress, timestamps, and configuration details throughout a defined lifecycle. Each run has an associated *Simulation Data* resource, which stores the inputs and outputs of a simulation run, capturing the values set into and read from the simulator model for traceability and integrity of the simulation data. [Simulation runs](/cdf/integration/guides/simulators/simulator_trigger) provide a historical record of the simulations performed for you to analyze and compare different runs, track changes over time, and make informed decisions based on the simulation results. [Simulator Logs](/api-reference/concepts/20230101/simulator-logs) provide detailed tracking of all system activities, including simulation runs, model parsing operations, and generic connector activities. Each log entry includes timestamp information, severity level, detailed messages, and any error information. These logs are essential for monitoring system health, debugging issues, auditing operations, and understanding system behavior. The logging system supports different severity levels to control the amount of detail captured, making it easier to focus on relevant information during troubleshooting or analysis. # Configure simulation routines Source: https://docs.cognite.com/cdf/integration/guides/simulators/simulator_routines Learn how to configure simulation routines with scheduling, data preprocessing, input/output mapping, and execution steps for running simulations. Simulator routines can have multiple revisions to track changes and evolve the routine. Each model can have multiple routines, each performing different tasks, such as calculating optimal operation set points, forecasting production, and benchmarking asset performance. Simulator routines can be considered as "folders" that organize different versions of simulation instructions. Each routine represents a specific objective you want to achieve with your model. For example, a well model might have separate routines for daily production optimization, monthly performance forecasting, equipment sizing studies, and what-if scenario analysis. Each routine revision is immutable. Once it's created, you can't modify the configuration or script. When you need to make changes, create a new revision. This ensures you maintain a complete history of your simulation configurations and can track how your routines evolve over time. ## Before you start * Consider the input data to the simulator. These are the values set into the simulator model. You can use constant values or time series, such as data from process sensors. For time series, make sure to consider [Data sampling](/cdf/integration/guides/simulators/about_data_sampling). * Plan the execution steps to issue to the simulator. Each simulator may support different steps, which are defined in the `stepFields` property of the simulator resource in CDF. * Decide which output data to read from the simulator. These are the values read from the simulation model. All connectors provided by Cognite support unit management, so you can define the units of the input and output values. The script in a routine determines the order of how the connector sets the input, issues commands, and fetches outputs. ## Routine revision structure * **Identification** uniquely identifies the revision and links it to its parent routine. * **Configuration**: * **Schedule** defines if or when the routine should run automatically. * **Data sampling** controls how time series data is processed. * **Input configuration** specifies values to set in the simulator. * **Output configuration** defines values to read from the simulator. * **Script** is an ordered list of stages and steps that define the execution sequence. ## Configure input **Constant inputs** are values that remain constant across simulation runs for a given routine revision. You can override these values when you trigger individual simulation runs. Unit specification is optional for numeric types. This input type is useful for parameters that rarely change but might need occasional adjustments. Valid value types: * `STRING` for text values. * `DOUBLE` for numeric values. * `STRING_ARRAY`for arrays of text. * `DOUBLE_ARRAY`for arrays of numbers. **Time series inputs** read values from CDF time series data points. You can override these values when you trigger individual simulation runs. Only the `DOUBLE` value type is valid. This input type is useful for dynamic parameters that typically come from sensors. The processing of this input depends on the data sampling configuration: * Latest value if data sampling is disabled. * Aggregated values based on sampling settings if this is enabled. ### Configuration parameters **Common** parameters for both input types: | Parameter | Description | | - | - | | `referenceId` | A unique identifier used to link the input with the script steps. | | `name` | A human-readable description. | | `unit` | Optional, but required to specify the unit of measure for numeric values. | | `saveTimeseriesExternalId` | Stores input values to time series data points. This is only used for the `DOUBLE` value type. This parameter is optional. | **Time series input** parameters: | Parameter | Description | | - | - | | `sourceExternalId` | References the source time series to read from. | | `aggregate` | Specifies how to aggregate values within the sampling window. | ## Configure output Outputs define the values to read from the simulator. Each output requires these parameters: | Parameter | Description | | - | - | | `referenceId` | Links the output with script steps. | | `name` | A human-readable description. | | `valueType` | The same options as inputs. | | `unit` | Enable unit awareness and conversion for numeric values. This parameter is optional. | | `saveTimeseriesExternalId` | Stores output values to time series data points. This is only used for the `DOUBLE` value type. This parameter is optional. | ## Script The script defines the execution sequence using ordered stages and steps. Each stage can contain multiple steps, and all steps within a stage execute sequentially. Steps are arranged into stages for better organization, and each stage and step has an `order` field that determines the execution sequence. Use the `description` fields to provide documentation for each step or stage. The step types and syntax vary by simulator, and the simulator resource in CDF defines available step types and syntax. ## Example This example shows a simple routine configuration for a system calculation using a PROSPER simulator. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "SystemCalculation-LER-WELL-A2-v1", "routineExternalId": "SystemCalculation-LER-WELL-A2", "configuration": { "schedule": { "enabled": true, "cron_expression": "0 * * * *" }, "dataSampling": { "enabled": true, "validationWindow": 60, "samplingWindow": 60, "granularity": 1 }, "logicalCheck": [ { "enabled": true, "timeseriesExternalId": "AP07-019THP-PSI2120.PV", "aggregate": "average", "operator": "ge", "value": 80 } ], "steadyStateDetection": [], "inputs": [ { "name": "RateMethod", "value": "2", "valueType": "STRING", "referenceId": "RateMethod", }, { "name": "Tubing Head Pressure", "referenceId": "THP", "unit": { "name": "psig", "quantity": "Pressure" }, "sourceExternalId": "AP07-019THP-PSI2120.PV", "aggregate": "average", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-INPUT-THP" }, { "name": "ClearSensDB", "value": "1", "valueType": "STRING", "referenceId": "ClearSensDB", } ], "outputs": [ { "name": "Gas Flowrate", "referenceId": "GasRate", "unit": { "name": "MMscf/day", "quantity": "Gas Rate" }, "valueType": "DOUBLE", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-OUTPUT-GasRate" }, { "name": "Oil Flowrate", "referenceId": "OilRate", "unit": { "name": "STB/day", "quantity": "Liquid Rate" }, "valueType": "DOUBLE", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-OUTPUT-OilRate" }, { "name": "Water Flowrate", "referenceId": "WatRate", "unit": { "name": "STB/day", "quantity": "Liquid Rate" }, "valueType": "DOUBLE", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-OUTPUT-WatRate" }, { "name": "Liquid Flowrate", "referenceId": "LiqRate", "unit": { "name": "STB/day", "quantity": "Liquid Rate" }, "valueType": "DOUBLE", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-OUTPUT-LiqRate" }, { "name": "Bottom Hole Pressure", "referenceId": "BHP", "unit": { "name": "psig", "quantity": "Pressure" }, "valueType": "DOUBLE", "saveTimeseriesExternalId": "SystemCalculation-LER-WELL-A2-OUTPUT-BHP" } ] }, "script": [ { "order": 1, "description": "Set Inputs", "steps": [ { "order": 1, "stepType": "Set", "description": "Set RateMethod to Automatic", "arguments": { "referenceId": "RateMethod", "address": "PROSPER.ANL.SYS.RateMethod" } }, { "order": 2, "stepType": "Set", "arguments": { "referenceId": "THP", "address": "PROSPER.ANL.SYS.Pres" } } ] }, { "order": 2, "description": "Perform Calculation", "steps": [ { "order": 1, "stepType": "Set", "description": "Clear table with sensitivity variables", "arguments": { "referenceId": "ClearSensDB", "address": "PROSPER.ANL.SYS.Sens.SensDB.Clear" } }, { "order": 2, "stepType": "Command", "arguments": { "address": "PROSPER.ANL.SYS.CALC" } } ] }, { "order": 3, "description": "Set Outputs", "steps": [ { "order": 1, "stepType": "Get", "arguments": { "referenceId": "GasRate", "address": "PROSPER.OUT.SYS.Results[0].Sol.GasRate" } }, { "order": 2, "stepType": "Get", "arguments": { "referenceId": "OilRate", "address": "PROSPER.OUT.SYS.Results[0].Sol.OilRate" } }, { "order": 3, "stepType": "Get", "arguments": { "referenceId": "WatRate", "address": "PROSPER.OUT.SYS.Results[0].Sol.WatRate" } }, { "order": 4, "stepType": "Get", "arguments": { "referenceId": "LiqRate", "address": "PROSPER.OUT.SYS.Results[0].Sol.LiqRate" } }, { "order": 5, "stepType": "Get", "arguments": { "referenceId": "BHP", "address": "PROSPER.OUT.SYS.Results[0].Sol.BHP" } } ] } ] } ``` ### Explanation #### Routine revision identification The routine revision is uniquely identified by `externalId ("SystemCalculation-LER-WELL-A2-v1")` and is associated with a parent routine ID, `"routineExternalId": "SystemCalculation-LER-WELL-A2"`, which is associated with a particular simulator (`"PROSPER"`) and a simulator model (`"LER-WELL-A2"`). #### Schedule configuration The routine revision is configured to run on a schedule, as indicated by the `"enabled": true` in the `"schedule"` section. The `"cronExpression": "0 * * * *"` means that the routine will run at the beginning of every hour. #### Data sampling [Data sampling](/cdf/integration/guides/simulators/about_data_sampling) is enabled for this routine. `validationWindow` and `samplingWindow` are both set to 60 minutes, with a granularity of 1 minute. This means that the connector will look for valid data within the last hour, sampling data every 1 minute. #### Logical check A logical check is configured for the tubing head pressure. It checks if the average value of time series `"AP07-019THP-PSI2120.PV"` is greater than `80`. This ensures that the simulation only runs when the tubing head pressure is above a certain threshold. #### Input configuration The routine revision defines three inputs for the simulation. Each input is assigned a reference ID for use in the simulation script. Some inputs are configured to save their values to time series for later analysis. | Input | Description | | - | - | | **RateMethod** | A constant value of `"2"`. | | **Tubing Head Pressure** | Sourced from the `"AP07-019THP-PSI2120.PV"` time series, using the average aggregate, with data stored in `"psig"`. | | **ClearSensDB** | A constant value of `"1"`. | #### Output configuration The routine revision defines five outputs from the simulation. | Input | Description | | - | - | | **Gas Flowrate** | In `"MMscf/day"`, to be saved to a time series. | | **Oil Flowrate** | In `"STB/day"`, to be saved to a time series. | | **Water Flowrate** | In `"STB/day"`, to be saved to a time series. | | **Liquid Flowrate** | In `"STB/day"`, to be saved to a time series. | | **Bottom Hole Pressure** | In `"psig"`, to be saved to a time series. | #### Simulation script The script section defines the steps for running the simulation: The script sets all the input values into the simulation model. The system calculation command, `"PROSPER.ANL.SYS.CALC"` is issued to run the simulation. The script retrieves the output values from the simulation model. This is a configuration for a system calculation of well A2, taking into account real-time tubing head pressure data and some PROSPER configuration flags. The simulation calculates the resulting production conditions, which can be used for various purposes such as virtual metering, production optimization, and asset health. # Run simulations Source: https://docs.cognite.com/cdf/integration/guides/simulators/simulator_trigger Learn how to trigger simulation runs manually, programmatically via API, or through scheduled execution in the simulation routine configuration. ## Before you start For a simulation to run, you need * A simulator model with at least one revision. * A simulator routine with at least one revision. * An active connector associated with your routine. ## Trigger runs manually Navigate to **Data fusion** > **Build solutions** in the Cognite Data Fusion (CDF) user interface and initiate a simulation run directly by selecting ▶ on a simulator routine. This is typically used for testing routines or running one-off simulations. By default, manual triggers use the latest model and routine revisions, and `runType` is set to **manual**. Set `runTime` to the trigger moment. ## Trigger runs programmatically Use the [Simulators API](/api-reference/concepts/20230101/simulators) to trigger simulations programmatically. Set up triggers using the API to integrate with external systems and workflows. `runType` is set to **external**. Using the API gives detailed control over execution parameters. ## Schedule trigger runs Configure routines to run automatically at specified times on a given frequency by setting up the `schedule` in the [simulation routine](/cdf/integration/guides/simulators/simulator_routines). Scheduled triggers use `cron` expressions to define precise timing. Scheduled triggers use the latest model and routine revisions, and `runType` is set to **scheduled**. If the connector is offline when the simulation is scheduled to run, the simulation is skipped. The connector only runs future schedules. Consider this limitation when using the native scheduling feature. ## Configuration parameters ### Resource selection | Parameter | Description | | - | - | | Default behavior | The simplest approach is to specify only the `routineExternalId`, which will use the latest revisions of both the routine and its associated model. | | Version control | For more control, you can specify both `routineRevisionExternalId` and `modelRevisionExternalId` to run simulations using specific historical versions of models and routines. This enables you to simulate past scenarios using model and routine configurations that were valid at that time. | ### Timing controls * **Run-time**: Define when the simulation should take place. For manual and API triggers, if not specified, defaults to the current time when the trigger is initiated. For scheduled runs, it matches the schedule time. ### Input overrides You can override any input defined in the routine: * Both constant values and time series references can be overridden * Numerical inputs accept values in any compatible unit of the same quantity * Overrides must maintain the same value type as defined in the routine revision * All inputs (constant, sampled, and overridden) are stored in the simulation data ### Execution settings * **Queue parameter**: Set to `true` to ensure simulation runs are created even when the associated connector is offline. * **Log severity**: Control the level of detail in execution logs, which is particularly useful for troubleshooting. ## Simulation status lifecycle Understanding how a simulation progresses through its lifecycle helps you track and manage your simulations effectively. Each simulation goes through three distinct phases from creation to completion. By default, when you first trigger a simulation run, the API creates the simulation run resource, assigns a unique `id` and `logId` for tracking purposes, records the exact creation time, and sets `status` to **ready**. The simulation run then waits in this state until a connector is ready to process it. If you enable **[Simulation run load balancing](/cdf/integration/guides/simulators/connectors#connector)** (*[preview](/cdf/product_feature_status)*) in your CDF project, runs, scheduled by CDF, wait in a shared **FIFO** queue (first in, first out). **Connectors** poll CDF for assigned runs and apply timeouts to stale runs. The `status` of the runs is **queued** until CDF assigns them to an available healthy connector for execution. CDF then sets the `status` of runs that are eligible for assignment to **ready** before execution starts. When the connector begins processing the simulation run, `status` is set to **running**. The connector executes the simulation according to the configured routine. The connector performs one last update to `lastUpdatedTime`, and sets `status` to **success** or **failure**. The simulation logs and results become available for analysis. Note that if the simulation fails, output data may be incomplete or missing, but logs can be consulted for troubleshooting. ## Accessing results Once a simulation completes, there are different resources which might be of interest: ### Simulation data The [simulation data](/api-reference/concepts/20230101/simulation-runs) endpoint provides the complete record of each simulation run. This includes all input values used during the simulation, whether they were constant values, overrides, or sampled from time series and the outputs generated during the simulation. This data serves as the primary source of truth for simulation results and provides complete traceability of the simulation process, including both the values and their associated units. ### Time series data For numerical values that have been configured to be saved as time series, the connector automatically stores both inputs and outputs as time series data points. Each data point has the timestamp of the simulation time, which lets you track trends across multiple simulation runs and compare results from different configurations. This integration with time series data makes it easy to analyze simulation results alongside other operational data in your CDF project. ### Execution logs The [execution logs](/api-reference/concepts/20230101/simulator-logs) provide detailed information about how your simulation ran, accessible using the simulation's `logId`. The logs capture the step-by-step progression of the simulation, with the level of detail determined by your configured log severity. Use the logs for troubleshooting failed simulations and understanding how your simulation executed. ## Important timestamps | Timestamp | Description | | - | - | | **Run Time** | Acts as the reference point for all data preprocessing activities and determines which input data will be considered for the simulation. | | **Simulation Time** | Represents when the input data was actually sampled and is used to index your results. Read more about how this timestamp is calculated in the [data sampling guide](/cdf/integration/guides/simulators/about_data_sampling). | | **Created Time** | When simulation run resource is created by the API. | | **Last Updated Time** | Changes throughout the simulation's lifecycle to track its progress through different stages. | ## Best practices **Model management** Maintaining accurate and up-to-date simulation models is important to obtain representative and reliable results. Keep your models calibrated with any changes to your physical assets, and use version control to track modifications over time. Document all changes to models, including the reasoning behind updates, to maintain a clear history of model evolution. **Resource optimization** Effective resource management requires balancing simulation frequency with available computational resources and license usage. Consider how often you really need updated results and adjust run schedules accordingly. For complex simulations or high-frequency requirements, consider distributing routines across multiple connectors to optimize resource usage and ensure reliable execution. **Data quality** Implementing robust data validation methods is also important for representative and reliable simulation results. Configure appropriate validation rules to ensure input data meets quality standards before running simulations. Regularly monitor input data quality and validate simulation results against expected ranges and patterns. Use the [data validation methods](/cdf/integration/guides/simulators/about_data_sampling) provided by CDF to automatically check for data quality issues. **Result management** Be aware that simulation runs and logs are subject to retention policies. See the [API documentation](/api-reference/concepts/20230101/simulators) for current limits. If you need to maintain permanent records of simulation runs and their associated logs, establish a data workflow to copy data into a permanent storage before the retention period expires. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/simulators/troubleshooting Find solutions to common simulator integration issues including scheduling, data validation, performance, and API rate limiting. ## The simulation doesn't start on schedule Check the connector status and ensure it's running. Verify that the routine's schedule is correctly configured and enabled. Check simulator logs for errors or warnings. ## Simulation fails due to invalid input data Review your data validation settings. Try adjusting the validation window or modify the logical checks to ensure valid data is available for the simulation. If simulations consistently fail to find valid conditions, consider relaxing the validation criteria. Review historical data to understand typical process behavior and adjust validation settings accordingly. ## Improve performance Consider optimizing your routines, distributing load across multiple connectors, and ensuring your simulation models are as efficient as possible. ## Simulations failing with license-related errors Most commercial simulators rely on local or server licenses to run simulations. The machine hosting the simulator must have access to a valid license. If licenses are shared across a network, make sure there are sufficient licenses available for connectors to use when triggering simulations. Consider providing dedicated licenses for simulation connectors to ensure stable operations. ## Simulations failing with "No data found for Input Time Series" When using input time series, ensure reliable data availability. Select an appropriate aggregate function depending on the type of process sensor. Consider disabling data sampling to use the latest datapoints if no up-to-date data is available. ## Scheduled simulations are being skipped If scheduled simulations are missing, this could be due to connector downtime during the scheduled execution time. Connectors only execute future schedules, so any simulations scheduled during connector downtime (maintenance, restarts, etc.) will be skipped and not retroactively executed. For high-availability requirements, consider these alternatives: Use Cognite Functions or Data workflows for scheduling instead of the connector's native scheduling mechanism. These cloud-based scheduling features will: * Track scheduled runs even when the connector is temporarily offline. This applies when simulation runs are triggered with `queue` equal to `true`. * Allow to edit the schedule configuration without creating a new routine revision. Implement a recovery mechanism using Cognite Functions to: * Detect skipped simulation runs. * Trigger retrospective simulations using the Simulators API by setting `runTime` to the skipped simulation schedule timestamp. ## Model revision create fails with a resource limit error If you see `Resource limit exceeded for simulator model revision. Max 200 per model.` (HTTP 422), the model has 200 revisions and the create request did not set `deleteOldest` to `true`. See [Model revision limit](/cdf/integration/guides/simulators/simulator_api#model-revision-limit). ## HTTP 429 (Too many requests) response status code Simulator integration endpoints return the HTTP 429 (too many requests) response status code when the back-end capacity limit is exceeded. Throttling can happen if a user (a person or a machine) sends more than the allocated number of requests in a given amount of time. The limit is 1000 requests per minute for the Simulators API. Make sure all automations that send requests to the Cognite API don't exceed this rate limit. Apply limits to your queries to avoid fetching too much data at once. If you have a large number of queries, consider breaking them into smaller chunks and spreading them over a period of time. This approach to data fetching is closer to "streaming" rather than "batch". If you have simulation results from multiple simulations and want to display these in a dashboard, consider storing the aggregated data in a materialized view and querying that instead. For example, this can be a time series, or a specifically designed data model. While direct consumption from the Simulators API is possible, users must consider the 1000 requests per minute limit. For use cases that may exceed this limit, we recommend materializing the data into a consumption-optimized storage. You can implement a background job using Cognite Functions and/or Data workflows to efficiently stream data from the Simulators API to your desired materialized storage within CDF. This approach ensures: * Efficient data access for high-throughput applications * Respect for API rate limits through controlled data streaming * Optimized storage for frequent data retrieval Use a retry strategy based on truncated exponential backoff with jitter to handle sessions with HTTP response code 429. [Learn more about API rate limits](/api-reference/concepts/20230101/rate-limits#simulators). # Data integration overview Source: https://docs.cognite.com/cdf/integration/index Connect your industrial data sources to Cognite Data Fusion with extractors, connectors, and transformation tools to build efficient data pipelines. Choose which data sources to integrate first and sequence your ingestion pipelines before you start building. Part of the Deploy tab. Explore the three main components of data pipelines in CDF: integration, transformation, and contextualization. Learn how extractors connect to source systems and stream data to CDF with minimal transformation. Discover how to reshape, enrich, and analyze data quality using built-in transformation tools. Connect and enrich industrial data by matching entities, parsing documents, and building relationships. ## Integrate and connect data Extract data from industrial systems, enterprise applications, and cloud sources using prebuilt or custom extractors. Integrate simulators, ETL tools, and enterprise systems with bidirectional data synchronization. Map, reshape, and enrich data from staging area to fit your target data models. Query Microsoft Fabric OneLake Delta tables without staging a full copy in CDF RAW using Transformations. Match entities, create interactive diagrams, and extract structured data from documents. Create on-premises integrations, complete the connection, and monitor tasks in Integrations (public preview). Register, configure, and monitor extraction pipelines with automatic failure notifications. Develop custom extractors and connectors using Python or .NET utilities for specialized requirements. # Import activities into Maintain Source: https://docs.cognite.com/cdf/maintain/guides/import_activities Learn how to import activities using the user interface (APM mode), Cognite Toolkit, or Transformations. Maintain supports three methods for importing activities: * **CSV upload**: Import activities using the user interface (APM mode only). * **Cognite Toolkit**: Ingest activities using [Cognite Toolkit](/cdf/deploy/cdf_toolkit/) data modeling (APM and IDM/CDM modes). * **Transformations**: Convert data from RAW tables or other sources using [CDF Transformations](/cdf/integration/) (APM and IDM/CDM modes). ## Import via CSV upload (APM mode) **Mode availability**: APM mode only. IDM/CDM mode doesn't support CSV import in the user interface. Activities imported via CSV can be visualized and modified as activities created in Maintain. The imported CSV file needs to conform to your project-specific template. To download **maintain-import-template.csv**: 1. Navigate to Activities and select Add activity. 2. Select Import activities. 3. Select Download template. The CSV template will only contain a single row of values representing the possible values you can provide for an imported activity. This row is called the header row and shouldn't be removed when populating the template. While the header **row** must not be removed, you can remove specific values from the header or change the order of the values. For example, if a certain field isn't relevant to import in a specific scenario, you can remove it from the header and the subsequent rows. This will result in the field being left blank on all imported activities. To import a single activity, add a new row to the template and enter the appropriate values separated by commas. Each value in the new row should contain the value you want to populate on the respective header row that was provided in the template. For example, if your template contains the following header row: `title,status,department`, create a CSV file with an additional row: ``` title,status,department Erect scaffolding,Done,Rigging ``` You'll create an activity with the title "Erect scaffolding", status "Done," and department "Rigging" when imported. ### Format date fields Some activity fields in **Maintain** represent dates and must have a specific format to be properly imported. Date fields can be identified in the CSV template by a special header value in the form `startTime (DD/MM/YYYY)`. In this example, `startTime` is the name of the field, while `DD/MM/YYYY` is the date format that needs to be used in subsequent rows of the CSV file. For example, importing the following file will result in the activity being created with `startTime` set to December 10th 2023. ``` title,startTime (DD/MM/YYYY) Erect scaffolding,10/12/2023 ``` ### Format boolean fields Some activity fields in **Maintain** represent boolean values, such as `true` or `false`. Boolean fields can be identified in the CSV template by a special header value in the form `isActive (true/false)`. These fields only have two valid values, `true` or `false`. For example, importing the following file will result in the activity being created with `isActive` set to `false`. ``` title,isActive (true/false) Erect scaffolding,false ``` ### Format values containing commas Due to the CSV format separating values using commas, be careful when an activity field also needs to contain a comma. **Maintain** processes CSV files according to the [RFC 4180 standard](https://www.rfc-editor.org/rfc/rfc4180), which requires values containing commas to be wrapped in double quotes. For example, importing the following file will result in the activity being created with `city` set to `Boston, MA`. ``` title,city Erect scaffolding,"Boston, MA" ``` ### Unset values You can leave the field blank on non-required activity fields to represent missing or irrelevant values for the specific activity. For example, importing the following file will result in the activity being created with `state` set to `New York`, but it will have no value in the `city` field. ``` title,city,state Erect scaffolding,New York ``` ### Connect an activity to an asset **Maintain** supports linking activities to CDF assets, which is used to contextualize the activity in 3D models, documents, PSN, and more. For **Maintain** to create the appropriate link to an asset in CDF, you must provide the CDF asset's external ID in the import template. If you have asset contextualization in your project, there will be a special `assetExternalId` field in the template that you need to populate with the external ID of the activity's asset. For example, when importing the following file and assuming a CDF asset exists with external ID `NY_BUILDING_23`, the activity will be contextualized to this asset when ingested into Maintain. ``` title,assetExternalId Erect scaffolding,NY_BUILDING_23 ``` If the provided external ID doesn't export, the user will see an error during import. End-users may not know the external ID of the asset to which the activity is linked. If it's necessary to provide end-users with a list of the possible assets and their external IDs, we recommend retrieving this data using CDF's [asset API](/api-reference/concepts/20230101/assets) or other data extractors. Once you have a populated CSV template, you can import it into **Maintain**: 1. Navigate to Activities > Add activity. 2. Select Import activities. 3. Select the Click to select CSV file to import field. 4. Select and upload the populated CSV file from your computer. 5. Verify that you imported the file by checking the File Inspector section. A green message should show the number of rows to be imported and if you have any warnings or errors. 6. If there are no errors, select Import to start the import process. 7. When the import succeeds, you'll see a green success message with the Show activities button. 8. Optional. Select Show activities to navigate to the imported activities and check that they look correct according to the CSV file. ## Import via Cognite Toolkit **Mode availability**: APM and IDM/CDM modes. This is the recommended approach for programmatic activity ingestion. Use the [Cognite Toolkit](/cdf/deploy/cdf_toolkit/) to ingest activities through CDF's data modeling capabilities. This method works for both deployment modes and provides version control and reproducible deployments. ### Prerequisites Before importing activities via the Cognite Toolkit, ensure you have: * A custom activity view configured in your data model space (or use the standard `CogniteMaintenanceOrder` view from `cdf_idm`) * The activity view mapped in your Maintain configuration under `viewMappings` * Access to the [Cognite Toolkit](/cdf/deploy/cdf_toolkit/) ### Configure the activity view mapping In your Maintain configuration, verify that the `viewMappings` section points to your activity view: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "viewMappings": { "activity": { "type": "view", "space": "cdf_idm", "version": "v1", "externalId": "CogniteMaintenanceOrder" } } ``` If you've created a custom activity view (for example, `APM_Activity` or `ExtendedActivity`), update the mapping to point to your custom view: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "viewMappings": { "activity": { "type": "view", "space": "maintain_schema_extention", "version": "v1", "externalId": "APM_Activity" } } ``` Learn more about [extending data models](/cdf/maintain/guides/config#extend-data-models) and [configuring view mappings](/cdf/maintain/guides/config#view-mappings). ### Ingest activities with the Cognite Toolkit Use the [Cognite Toolkit](/cdf/deploy/cdf_toolkit/) as the recommended approach for ingesting activities in IDM mode. If you haven't already, create Container, View, and Node YAML files for your activities. Follow the standard data modeling structure: * `APM_Activity.Container.yaml` - Defines the container schema * `APM_Activity.View.yaml` - Defines the view properties and relationships * `apmActivities.Node.yaml` - Contains your activity instances See the [Cognite Toolkit documentation](/cdf/deploy/cdf_toolkit/) for detailed examples and templates. Ensure your activity nodes include the required fields for Maintain: | Field | Description | Required | | - | - | - | | `externalId` | Unique identifier for the activity | Yes | | `title` | Activity title | Yes | | `status` | Activity status (for example, "Open", "In Progress", "Done") | Recommended | | `startTime` | Start date/time | Recommended | | `endTime` | End date/time | Recommended | | `rootLocation` | Root location identifier | Recommended | | `assetExternalId` | External ID of linked asset | Optional | | `mainAsset` | Direct relation to asset | Optional | Include additional fields based on your [Field Configuration](/cdf/maintain/guides/config#field-configuration) settings in Maintain. Deploy your activity data using Cognite Toolkit: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cdf deploy --env dev ``` Use the `--drop-data` flag if you need to replace existing activities: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cdf deploy --env dev --drop-data ``` Learn more about the [Cognite Toolkit deployment flags](/cdf/maintain/guides/config#critical-deployment-flags). 1. Sign in to [Maintain](https://maintain.cogniteapp.com). 2. Navigate to **Activities**. 3. Verify that your imported activities appear with the correct data. ### Ingest activities with Transformations **Mode availability**: APM and IDM/CDM modes. This is the recommended approach for bulk data. Alternatively, you can use [CDF Transformations](/cdf/integration/) to convert data from RAW tables or other sources into activity nodes. ### Troubleshooting If you encounter issues with activities after ingestion, see the table for causes and fixes. | Issue | Solution | | - | - | | Activities not visible | Verify the `viewMappings` configuration points to the correct space, view, and version | | Missing fields | Check that your Field Configuration includes all the fields you're ingesting | | Asset links broken | Ensure `assetExternalId` or `mainAsset` relations point to existing assets | | Wrong root location | Verify `rootLocation` matches your location configuration | # Plan activities Source: https://docs.cognite.com/cdf/maintain/guides/set_work_scope Create and manage plans to schedule activities, identify conflicts, and optimize work scope. The features described in this guide require specific configuration settings. If a feature isn't available in your Maintain environment, contact your administrator to enable it. Learn more about [configuring Maintain](/cdf/maintain/guides/config). Open Plans to plan and schedule activities and collaborate with coworkers. Open Activities to see all activities in your location. You can filter the list to find activities relevant to your plan. Create a plan with a list of activities to be performed during an event, such as a campaign. You can find all plans under the Plans tab and recently created and viewed plans on the home page. 1. Select Plan > + Create new plan. 2. Name your plan and select attributes. You can create an empty plan and add activities later or use other attributes to filter on activities you want to include in the plan. 3. Select Preview to view all the activities to be added to your plan, and select Apply to create the plan. Open Plan analysis to assess your plan, then identify possible conflicts and necessary actions by viewing your plan through multiple lenses. ### Analyze your plan Plan analysis shows the plan breakdown for activity attributes such as job type, material status, long lead items, vendors' requirements, hot work, and lifting operations. You can also see the status of work preparations, such as permits and work package documents. Plan Analysis requires [specific configuration](/cdf/maintain/guides/config#plan-analysis) including the `MAINTAIN_budget_analysis` feature flag and required operation data fields (`personHours`, `numberOfMainResource`). ### Identify conflicts and improvements Catch opportunities and avoid conflicts using the lenses: * Canvas: View all documents related to your plan, such as **engineering diagrams** (P\&IDs) and plot plans. Select a P\&ID to see all the activities in a plan overlaid on the engineering diagram. You can also annotate the diagram with text, color, and lines to create isolation plans to support your plan optimization. Add comments to collaborate with co-workers or operations on, for example, isolation plans. Learn more about [configuring the 2D viewer](/cdf/maintain/guides/config#2d-viewer) and [diagram parsing](/cdf/integration/guides/contextualization/parse_diagrams). * 3D: Add spatial context to see how activities and risks are placed in relation to each other. Zoom into the equipment to understand access, check the surroundings, investigate if you need more equipment, and locate other nearby activities. Learn more about [configuring 3D models](/cdf/maintain/guides/config#3d-and-digital-twin). If your environment has [grid zones](/cdf/maintain/guides/config#grid-zones) configured, you can navigate directly to specific zones on P\&IDs and 3D models from activities for easier spatial navigation. ### Discover opportunities Use Compare to identify opportunities. You can view and add activities outside of your plan or include them as a reference while scoping your work. 1. Select Compare > Activities not in my plan to find activities you want to add to your plan. The activities inside and outside the plan have different color codes in the lenses. 2. Optional. Filter activities to narrow down the search. 3. Select the activities you want to add to your plan. It can be helpful to compare the activities that occur in the same time frame, in the same location, with the same job type, etc. Use Group to analyze your campaign or shutdown in multiple ways, for example, by platform, deck level, grid level, and job type. * You can configure, rearrange, and add new activity attributes to the Group list. * The table view shows the result of your grouping. You can interact with and rearrange the grouped activities. Use Plan analysis to view how your plan is impacted as you add or remove activities; if you are overloaded or underloaded. If you are overloaded, remove an activity to see how it impacts the plan. You can also sort by order type, OLAFD date, and total hours. Use Budget and hourly optimization to create a baseline for comparison: POB, duration, non-productive hour %, and shift duration. The Budget and hourly optimization feature requires the `MAINTAIN_hour_utilization` feature flag and that your operation data includes `personHours`, `numberOfMainResource`, and `mainResource` fields. Learn more about [Plan Analysis configuration](/cdf/maintain/guides/config#plan-analysis). Create Layouts to save your preferred filters, grouping, and plan analysis. Learn more about [configuring layouts](/cdf/maintain/guides/config#layout). Maintain connects scheduling data from several source systems to create visibility around one schedule. You can comment and collaborate on the schedule with your co-workers in Maintain. Schedule editing requires additional configuration if this isn't turned on in your environment. The Gantt chart shows the activities on a timeline to identify the best sequence of activities. It's supported by color-coding that shows the material status and the trades. * Open the Gantt chart to sequence and schedule activities. * Add comments to collaborate with your co-workers. Learn more about [configuring the Gantt chart](/cdf/maintain/guides/config#gantt). Maintain automatically maps activities to functional locations on documents such as P\&IDs and plot plans. This mapping can be exported to PDF files to share your plan with colleagues and vendors. * Next to Add activity, select More options (⋮) > Export PDF to download the P\&IDs (with markup, colors, and isolation boundaries), with notes and activities in the plan, and share them with colleagues, vendors, and contractors. * You can export the activities list and Gantt chart to PDF. The Excel export is only for the table view. * Depending on the filter you use to view your activities, Documents lets you search for documents and tags available in that filter view that you can download as a PDF. Switch between lenses and tools to optimize your plan. # About Cognite Maintain Source: https://docs.cognite.com/cdf/maintain/index Cognite Maintain helps you plan, review, and improve your field activities for medium to long-term projects. The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. With Maintain, you can visualize activities and all related data to find potential issues or opportunities, create detailed **campaign plans** for events like shutdowns and turnarounds, discover **opportunity scopes** and reduce isolation frequency and production loss, then monitor the progress of your activities, make adjustments, and **analyze maintenance trends** over time. ## Get started Create and manage plans to schedule activities, identify conflicts, and optimize work scope. Upload activity data in bulk from external sources using CSV files. Access the Maintain application to start planning and reviewing your field activities. # Sign in to CDF Source: https://docs.cognite.com/cdf/sign-in To sign in to Cognite Data Fusion, you must provide your organization's name, enter your credentials, and specify which project you're signing in to. Navigate to [https://fusion.cognite.com](https://fusion.cognite.com). Enter your organization name and press Enter. Often, this is your company name without spaces or an abbreviation of your company name. **OR** Select an organization you've recently signed in to. If prompted, enter your work account credentials. Select the CDF project name. If you need to know what to enter to sign in, you can contact your internal help desk or the **CDF** **admin** for help. The CDF admin often gave you the link to sign in to CDF. If you need help finding out who the CDF admin is in your organization, contact [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new). ## Problems signing in? If you have checked that you have entered the correct information but are still not able to sign in, try these tips before you contact [support@cognite.com](mailto:support@cognite.com): ### Restart the browser A signed-in session may still be active on the browser. To deactivate this session, you must quit and restart the browser. Then, navigate to the application where you want to sign in. ### Clear the browser cache The browser may have cached an older version of the application. To load the latest version: Navigate to the application where you want to sign in. Reload the current page, ignoring cached content: * Windows: Ctrl + Shift + R * macOS: Cmd + Shift + R Proceed with the sign-in process. ### Clear browser cookies Sometimes, the cookies set by the identity provider (for example, Microsoft Entra ID, Amazon Cognito, Google Cloud Platform) are stale or can't be read. Follow the steps for your browser to clear the cookies: * [Google Chrome](https://support.google.com/answer/32050) * [Microsoft Edge](https://support.microsoft.com/en-us/help/4027947/windows-delete-cookies) # Subscribe to CDF Source: https://docs.cognite.com/cdf/sign-up Subscribe to Cognite Data Fusion (CDF) and connect to and use our cloud-based apps and services. To subscribe to Cognite Data Fusion, contact us [**here**](https://www.cognite.com/contact). When you subscribe, you'll manage a CDF **organization**. An organization is a collection of CDF **projects**. The data in one project is **completely isolated** from the data in other projects. Depending on your existing infrastructure and needs, you decide which **cluster** to run your CDF organization in. A cluster determines which **cloud provider** to use — Microsoft Azure, Amazon Web Services (AWS), or Google Cloud Platform (GCP). Each provider has [different clusters](/cdf/admin/clusters_regions#cognite-multi-tenant-clusters) to store data in your preferred **region**, such as North America, Asia, or Europe. Also, you can choose between a **multi-tenant** or a **dedicated** cluster: * In a **multi-tenant** cluster, you share cloud storage and computing resources with other Cognite tenants. * In a **dedicated** cluster, your organization uses cloud storage and computing resources exclusively. To request a dedicated cluster, contact your Cognite representative. You can configure your organization to use your existing **identity provider (IdP)** — for example, Microsoft Entra ID, Amazon Cognito, or Auth0 — to manage access to your CDF projects. Only users registered in your IdP can sign in to the organization and the projects. Access management concepts # Cognite documentation Source: https://docs.cognite.com/index Self-service product documentation for developers and technology professionals. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). Connect your development environment to Cognite's documentation through MCP server and SKILL.md integration. Build and deploy production-ready React applications in CDF. UI framework for CDF and Flows custom apps — foundations, primitives, and patterns. ## Flows and field operations Manage analysis, visualizations, AI agents, and field activities. Pre-built applications for industrial data analysis. Conversational AI agents for industrial workflows. Interactive data visualization and monitoring. InField and Maintain solutions. ## Knowledge graph and data integration Build the data foundation and core capabilities that power all Cognite Data Fusion (CDF) solutions. Connect with existing systems and data sources. Create unified knowledge graphs for your industrial data. Configure authentication and access control. ## What's new See all new features and enhancements in the latest Cognite Data Fusion releases. [**See all >**](/cdf/whatsnew) ## Deploy and operate Implement and manage your solutions in production environments. Best practices for building and implementing CDF solutions. Automate data processing and transformation. Infrastructure as code for CDF project management. Data quality, lineage, and compliance tools. ## Development tools APIs, SDKs, and developer resources for building on CDF. Learn how you can connect your developer tools to our [**MCP server**](https://docs.cognite.com/mcp) and from the dropdown menu on the top right of every article page. Get started with our comprehensive REST API. Full-featured Python library for CDF integration. Browser and Node.js support for web applications. .NET, Java, Rust, and specialized toolkits. Create, build, and deploy Flows custom apps from the terminal with @cognite/cli. Build and deploy production-ready React applications in CDF. UI framework for CDF and Flows custom apps — foundations, primitives, and patterns. # Cognite extractors overview Source: https://docs.cognite.com/cdf/integration/concepts/extraction/get_started Explore prebuilt extractors to stream and batch extract data from industrial and enterprise systems into Cognite Data Fusion (CDF). Extract time series, events, and asset data from source systems to CDF with prebuilt extractors or build custom solutions for specialized integration requirements. Extractors maintain read-only access and never modify original data in source systems. Understand how extractors stream or batch extract data to CDF staging area or services with no transformation of original data. Connect to OPC UA protocol to stream time series and events with real-time subscriptions and batch extract node hierarchies as assets. Stream time series from OSISoft PI Data Archive with parallel historical backfill and real-time data updates to CDF. Extract data from SQL Server, Oracle, PostgreSQL, and MySQL using ODBC drivers with batch extraction to CDF staging or time series. Connect to SAP systems through OData endpoints to batch extract master data and hierarchies into CDF staging area. Batch extract Asset Framework elements and attributes to build asset hierarchies in CDF staging area with complete metadata. Develop extractors in Python or .NET using Cognite SDK utilities with built-in configuration, state management, and logging. Configure serverless extractors in CDF with custom mappings and built-in functions without managing infrastructure. # About data extraction Source: https://docs.cognite.com/cdf/integration/concepts/extraction/index Understand how extractors connect to source systems and push data in its **original format** to Cognite Data Fusion using prebuilt extractors or custom solutions with Python and .NET utilities. The prebuilt extractors can **stream or extract data in batches** to the [CDF staging area](/cdf/integration/guides/extraction/raw_explorer) or directly to a [CDF service](/dev/concepts/resource_types) with little or no data transformation. The Cognite extractors only need read access to the source systems and never change the original data. To integrate data into a data model, you can: * Use Cognite's extractor and transformation components. * Use 3rd party extractor and transformation components. * Develop custom solutions. You can also ingest data into CDF using the [Cognite API](/api-reference/concepts/20230101/api-description) or integrate Extract, Transform, Load (ETL) tools with the [PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw) if your data is already in the cloud. Integration architecture ## Cognite extractors | Extractor | Description | | - | - | | [Cognite DB extractor](/cdf/integration/guides/extraction/db) | Connects to databases using ODBC, runs queries, and batch extracts data into the CDF staging area or directly to the CDF time series service. Make sure you have ODBC drivers for the databases you're connecting to. This extractor is available as both a Windows service and a standalone executable. | | [Cognite OPC UA extractor](/cdf/integration/guides/extraction/opc_ua) | Connects to the open OPC UA protocol and streams time series into the CDF time series service and events into the CDF events service. It batch extracts the OPC UA node hierarchy into the CDF staging area or as CDF assets and relationships. | | [Cognite PI extractor](/cdf/integration/guides/extraction/pi) | Connects to the PI Data archive and streams time series into the CDF time series service. In parallel, the extractor ingests historical data (backfilling). | | [Cognite PI AF extractor](/cdf/integration/guides/extraction/pi_af) | Connects to the PI Asset Framework and batch extracts Asset Framework elements and all their attributes. The extractor then builds a tree for each element in the CDF staging area. | | [Cognite PI replace utility](/cdf/integration/guides/extraction/pi_replace) | Re-ingests time series to CDF. Use this utility if the PI extractor can't sync PI content due to invalid or missing data points. | | [Cognite Studio For Petrel extractor](/cdf/integration/guides/extraction/petrel_studio) | Connects to SLB Studio For Petrel and extracts the records to CDF as protobuf objects. | | [Cognite EDM extractor](/cdf/integration/guides/extraction/edm) | Connects to Landmark Engineers Data Model database and exports the data to the CDF staging area. | | [Cognite WITSML extractor](/cdf/integration/guides/extraction/witsml) | Connects to [WITSML](https://www.energistics.org/witsml-data-standards) data sources and sends the data to CDF. | | [Cognite OSDU extractor](/cdf/integration/guides/extraction/osdu) | Connects to [OSDU](https://osduforum.org) and sends the data to the CDF staging area. | | [Cognite File extractor](/cdf/integration/guides/extraction/file) | Connects to local file systems, cloud storage, and network sharing protocols to upload files into CDF. You can also use this extractor to connect to OpenText Documentum and Documentum D2. It batch extracts documents into the CDF files service. | | [Cognite SAP Extractor](/cdf/integration/guides/extraction/sap) | Connects to SAP through OData endpoints and batch extracts data into the CDF staging area. | ### Download installation files Navigate to Data fusion > Integrate > Extractors. Select the extractor best suited to your source system. Download the **Installation file** and, optionally, the PDF documentation. Extractor download page You can also start from Data fusion > Integrate > Integrations (public preview). Select Set up, choose the extractor, then download and install from the integration. After the extractor connects to CDF, open the integration to monitor tasks and manage configuration. See [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration). ## Custom extractors You can create custom extractors if your source system can't use any of the prebuilt extractors. Select **Create a custom extractor** to find Python and .NET utils packages and SDKs to get started. ### Cognite Python extractor-utils SDK Use the [Python extractor-utils SDK](https://github.com/cognitedata/python-extractor-utils), an extension of the [Cognite Python SDK](/dev/sdks/python), to develop extractors in Python. For more information, see the [Python extractor-utils SDK documentation](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest). ### Cognite .NET extractor utilities code Use the [.NET extractor](https://github.com/cognitedata/dotnet-extractor-utils) to develop extractors in C#. For more information, see the [.NET utils documentation](https://cognitedata.github.io/dotnet-extractor-utils/tutorials/intro.html). # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/configuration/db Complete reference for configuring the Cognite DB extractor, including authentication, queries, and destination settings. To configure the DB extractor, you must create a configuration file. The file must be in YAML format. You can set up extraction pipelines to use versioned extractor configuration files stored in the cloud. ## Using values from environment variables The configuration file allows substitutions with environment variables. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite: secret: ${COGNITE_CLIENT_SECRET} ``` This will load the value from the `COGNITE_CLIENT_SECRET` environment variable into the `cognite/secret` parameter. You can also do string interpolation with environment variables, for example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN} ``` Implicit substitutions only work for unquoted value strings. For quoted strings, use the `!env` tag to activate environment substitution: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: !env 'http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}' ``` ## Using values from Azure Key Vault The DB extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `azure-keyvault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the Azure CLI. For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | Example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Base configuration object | Parameter | Type | Description | | - | - | - | | `version` | either string or integer | Configuration file version | | `type` | either `local` or `remote` | Configuration file type. Either `local`, meaning the full config is loaded from this file, or `remote`, which means that only the `cognite` section is loaded from this file, and the rest is loaded from extraction pipelines. Default value is `local`. | | `cognite` | object | The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | | `logger` | object | The optional `logger` section sets up logging to a console and files. | | `metrics` | object | The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a Prometheus pushgateway, but you can also send metrics as time series in the CDF project. | | `queries` | list | List of queries to execute | | `databases` | list | List of databases to connect to | | `extractor` | object | General extractor configuration | ## `cognite` Global parameter. The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | Parameter | Type | Description | | - | - | - | | `project` | string | Insert the CDF project name. | | `idp-authentication` | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | | `data-set` | object | Enter a data set the extractor should write data into | | `extraction-pipeline` | object | Enter the extraction pipeline used for remote config and reporting statuses | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | `timeout` | integer | Enter the timeout on requests to CDF, in seconds. Default value is `30`. | | `external-id-prefix` | string | Prefix on external ID used when creating CDF resources | | `connection` | object | Configure network connection details | ### `idp-authentication` Part of `cognite` configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | Parameter | Type | Description | | - | - | - | | `authority` | string | Insert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | list | Enter a list of scopes requested for the token | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | `certificate` | object | Authenticate with a client certificate | #### `scopes` Part of `idp-authentication` configuration. Enter a list of scopes requested for the token Each element of this list should be a string. #### `certificate` Part of `idp-authentication` configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `data-set` Part of [`cognite`](#cognite) configuration. Enter a data set the extractor should write data into | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Enter the extraction pipeline used for remote config and reporting statuses | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `connection` Part of [`cognite`](#cognite) configuration. Configure network connection details | Parameter | Type | Description | | - | - | - | | `disable-gzip` | boolean | Whether or not to disable gzipping of json bodies. | | `status-forcelist` | string | HTTP status codes to retry. Defaults to 429, 502, 503 and 504 | | `max-retries` | integer | Max number of retries on a given http request. Default value is `10`. | | `max-retries-connect` | integer | Max number of retries on connection errors. Default value is `3`. | | `max-retry-backoff` | integer | Retry strategy employs exponential backoff. This parameter sets a max on the amount of backoff after any request failure. Default value is `30`. | | `max-connection-pool-size` | integer | The maximum number of connections which will be kept in the SDKs connection pool. Default value is `50`. | | `disable-ssl` | boolean | Whether or not to disable SSL verification. | | [`proxies`](#cognite.connection.proxies) | object | Dictionary mapping from protocol to url. | #### `proxies` Part of [`connection`](#cognite.connection) configuration. Dictionary mapping from protocol to url. ## `logger` Global parameter. The optional `logger` section sets up logging to a console and files. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Include the console section to enable logging to a standard output, such as a terminal window. | | [`file`](#logger.file) | object | Include the file section to enable logging to a file. The files are rotated daily. | | `metrics` | boolean | Enables metrics on the number of log messages recorded per logger and level. This requires `metrics` to be configured as well | ### `console` Part of [`logger`](#logger) configuration. Include the console section to enable logging to a standard output, such as a terminal window. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for console logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | ### `file` Part of [`logger`](#logger) configuration. Include the file section to enable logging to a file. The files are rotated daily. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for file logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | | `path` | string | **Required.** Insert the path to the log file. | | `retention` | integer | Specify the number of days to keep logs for. Default value is `7`. | ## `metrics` Global parameter. The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | Parameter | Type | Description | | - | - | - | | [`push-gateways`](#metrics.push-gateways) | list | List of prometheus pushgateway configurations | | [`cognite`](#metrics.cognite) | object | Push metrics to CDF timeseries. Requires CDF credentials to be configured | | [`server`](#metrics.server) | object | The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | ### `push-gateways` Part of [`metrics`](#metrics) configuration. List of prometheus pushgateway configurations Each element of this list should be a the push-gateways sections contain a list of metric destinations. | Parameter | Type | Description | | - | - | - | | `host` | string | Enter the address of the host to push metrics to. | | `job-name` | string | Enter the value of the `exported_job` label to associate metrics with. This separates several deployments on a single pushgateway, and should be unique. | | `username` | string | Enter the credentials for the pushgateway. | | `password` | string | Enter the credentials for the pushgateway. | | `clear-after` | either null or integer | Enter the number of seconds to wait before clearing the pushgateway. When this parameter is present, the extractor will stall after the run is complete before deleting all metrics from the pushgateway. The recommended value is at least twice that of the scrape interval on the pushgateway. This is to ensure that the last metrics are gathered before the deletion. Default is disabled. | | `push-interval` | integer | Enter the interval in seconds between each push. Default value is `30`. | ### `cognite` Part of [`metrics`](#metrics) configuration. Push metrics to CDF timeseries. Requires CDF credentials to be configured | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | **Required.** Prefix on external ID used when creating CDF time series to store metrics. | | `asset-name` | string | Enter the name for a CDF asset that will have all the metrics time series attached to it. | | `asset-external-id` | string | Enter the external ID for a CDF asset that will have all the metrics time series attached to it. | | `push-interval` | integer | Enter the interval in seconds between each push to CDF. Default value is `30`. | | [`data-set`](#metrics.cognite.data-set) | object | Data set the metrics will be created under | #### `data-set` Part of [`cognite`](#metrics.cognite) configuration. Data set the metrics will be created under | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `server` Part of [`metrics`](#metrics) configuration. The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | Parameter | Type | Description | | - | - | - | | `host` | string | Host to run the prometheus server on. Default value is `0.0.0.0`. | | `port` | integer | Local port to expose the prometheus server on. Default value is `9000`. | ## `queries` Global parameter. List of queries to execute Each element of this list should be a description of a SQL query against a database | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Enter the name of the database to connect to. This must be one of the database names configured in the `databases` section. | | `name` | string | **Required.** Enter a name of this query that will be used for logging and tagging metrics. The name must be unique for each query in the configuration file. | | `query` | string | **Required.** SQL query to execute. Supports interpolation with `{incremental_field}` and `{start_at}` | | [`destination`](#queries.destination) | configuration for either RAW, Events, Assets, Time series, Sequence, Files, Nodes or Edges | **Required.** The destination of the data in CDF.

**Examples:**
`{'destination': {'type': 'raw', 'database': 'my-database', 'table': 'my-table'}}`
`{'destination': {'type': 'events'}}`
`{'destination': {'type': 'time_series', 'destination_mode': 'cdm', 'data-model': {'space': 'my-space'}}}`
`{'destination': {'type': 'time_series', 'destination_mode': 'classic'}}` | | `primary-key` | string | Insert the format of the row key in CDF RAW. This parameter supports case-sensitive substitutions with values from the table columns. For example, if there's a column called index, setting `primary-key: row_{index}` will result in rows with keys `row_0`, `row_1`, etc. This is a required value if the destination is a `raw` type.

**Example:**
`row_{index}` | | `incremental-field` | string | Insert the table column that holds the incremental field. Include to enable incremental loading, otherwise the extractor will default to a full run every time. To use incremental load, a state store is required | | `initial-start` | either string, number or integer | Enter the `{start_at}` for the first run. Later runs will use the value stored in the state store. Will only be used on the initial run, subsequent runs will use the stored state. Required when incremental-field is set. | | [`schedule`](#queries.schedule) | configuration for either Fixed interval or CRON expression | Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise.

**Examples:**
`{'schedule': {'type': 'interval', 'expression': '1h'}}`
`{'schedule': {'type': 'cron', 'expression': '0 7-17 * * 1-5'}}` | | `collection` | string | Specify the collection on which the query will be executed. This parameter is mandatory when connecting to `mongodb` databases. | | `container` | string | Specify the container on which the query will be executed. This parameter is mandatory when connecting to `cosmosdb` databases. | | `sheet` | string | Specify the sheet on which the query will be executed. This parameter is mandatory when connecting to `spreadsheet` files. | | `skip_rows` | string | Specify the number of rows to be skipped when reading a spreadsheet. This parameter is optional when connecting to `spreadsheet` files. | | `has_header` | string | Specify if the extractor should skip the file header while reading a spreadsheet. This parameter is optional when connecting to `spreadsheet` files. | | `parameters` | string | Specify the parameters to be used when querying to AWS DynamoDB. This parameter is mandatory when connectong to `dynamodb` databases. | ###
`destination` Part of [`queries`](#queries) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#queries.destination.raw) * [Events](#queries.destination.events) * [Assets](#queries.destination.assets) * [Time series](#queries.destination.time_series) * [Sequence](#queries.destination.sequence) * [Files](#queries.destination.files) * [Nodes](#queries.destination.nodes) * [Edges](#queries.destination.edges) **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} destination: type: raw database: my-database table: my-table ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} destination: type: events ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} destination: type: time_series destination_mode: cdm data-model: space: my-space ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} destination: type: time_series destination_mode: classic ``` #### `raw` Part of [`destination`](#queries.destination) configuration. The raw destination writes data to the CDF staging area (RAW). The raw destination requires the `primary-key` parameter in the query configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | **Required.** Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | **Required.** Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | #### `events` Part of [`destination`](#queries.destination) configuration. The `events` destination inserts the resulting data as CDF events. The events destination is configured by setting the `type` parameter to `events`. No other parameters are required. To ingest data into a events, the query must produce columns named * `externalId` In addition, columns named * `startTime` * `endTime` * `description` * `source` * `type` * `subType` may be included and will be mapped to corresponding fields in CDF events. Any other columns returned by the query will be mapped to key/value pairs in the `metadata` field for events. | Parameter | Type | Description | | - | - | - | | `type` | always `events` | Type of CDF destination, set to `events` to write data to events. | #### `assets` Part of [`destination`](#queries.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. The assets destination is configured by setting the `type` parameter to `assets`. No other parameters are required. To ingest data into a assets, the query must produce columns named * `name` In addition, columns named * `externalId` * `parentExternalId` * `description` * `source` may be included and will be mapped to corresponding fields in CDF assets. Any other columns returned by the query will be mapped to key/value pairs in the `metadata` field for assets. | Parameter | Type | Description | | - | - | - | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to assets. | #### `time_series` Part of [`destination`](#queries.destination) configuration. The `time_series` destination inserts the resulting data as data points in time series. The time series destination is configured by setting the `type` parameter to `time_series`. No other parameters are required. To ingest data into a time series, the query must produce columns named * `externalId` * `timestamp` * `value` In addition, include a column called `status` to give the datapoint a status code. Statuses include a category, and an optional comma-separated list of modifyer flags. Some examples for status codes include `Good` (which is assumed if status is omitted), `UNCERTAIN, HIGH` and `bad`. The extractor will insert data points into time series identified by the `externalId` column. If a time series does not exist, the extractor will create a minimal time series with only an external ID and the `isString` property inferred from the type of first data point processed for that time series. All other time series attributes need to be added separately. | Parameter | Type | Description | | - | - | - | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to time series. | | `destination_mode` | either `cdm` or `classic` | Mode of the db extractor. Can be 'cdm' for Data models or 'classic' for legacy Timeseries. | | [`data-model`](#queries.destination.time_series.data-model) | object | Defines Data Model mapping information, including the target Space and Data Model. This property is only applicable when the destination `type` is set to `timeseries`. | ##### `data-model` Part of [`time_series`](#queries.destination.time_series) configuration. Defines Data Model mapping information, including the target Space and Data Model. This property is only applicable when the destination `type` is set to `timeseries`. | Parameter | Type | Description | | - | - | - | | `space` | string | Enter the CDF space name. An error will occur if the specified space does not exist. | #### `sequence` Part of [`destination`](#queries.destination) configuration. The `sequence` destination writes data to a CDF sequence. The column set of the query result will determine the columns of the sequence. The result must include a column named `row_number`, which must include an integer indicating which row number in the sequence to ingest the row into. | Parameter | Type | Description | | - | - | - | | `type` | always `sequence` | Type of CDF destination, set to `sequence` to write data to a sequence. | | `external-id` | string | **Required.** Configured sequence external ID | | `value-types` | either `convert`, `drop` or `assert` | How types are converted into the expected types in CDF. Convert attempts to make a conversion, which may fail. Drop drops the row if there is a mismatch. Assert fails the query if the types do not match. Default value is `convert`. | #### `files` Part of [`destination`](#queries.destination) configuration. The `files` destination inserts the resulting data as CDF files. The files destination is configured by setting the `type` parameter to `files`. No other parameters are required. To ingest data into a files, the query must produce columns named * `name` * `externalId` * `content` `content` will be treated as binary data and uploaded to CDF files as the content of the file In addition, columns named * `source` * `mimeType` * `directory` * `sourceCreatedTime` * `sourceModifiedTime` * `asset_ids` may be included and will be mapped to corresponding fields in CDF files. Any other columns returned by the query will be mapped to key/value pairs in the `metadata` field for files. | Parameter | Type | Description | | - | - | - | | `type` | always `files` | Type of CDF destination, set to `files` to write data to CDF files. | | `content-column` | string | Column used as file content. Default value is `content`. | | `destination-mode` | either `cdm` or `classic` | Mode of the db extractor. Can be 'cdm' for Data models or 'classic' for legacy Files. | | [`data-model`](#queries.destination.files.data-model) | object | Defines Data Model mapping information, including the target Space and Data Model. This property is only applicable when the destination `type` is set to `files`. | ##### `data-model` Part of [`files`](#queries.destination.files) configuration. Defines Data Model mapping information, including the target Space and Data Model. This property is only applicable when the destination `type` is set to `files`. | Parameter | Type | Description | | - | - | - | | `space` | string | Enter the CDF space name. An error will occur if the specified space does not exist. | #### `nodes` Part of [`destination`](#queries.destination) configuration. The `nodes` destination inserts the resulting data into nodes containers. The nodes destination is configured by setting the `type` parameter to `nodes`. You have to add data-model. To ingest data into a nodes, the query must produce columns named * `externalId` In addition, other columns (according to data model fields) may be included and will be mapped to corresponding fields in CDF nodes. Any other columns returned by the query will not be ingested or throw an error. | Parameter | Type | Description | | - | - | - | | `type` | always `nodes` | Type of CDF destination, set to `nodes` to write data to CDF nodes. | | `space` | string | Enter the CDF space name. An error will occur if the specified space does not exist. | | [`view`](#queries.destination.nodes.view) | object | Enter the CDF view/container configuration. This is required to write to nodes. | ##### `view` Part of [`nodes`](#queries.destination.nodes) configuration. Enter the CDF view/container configuration. This is required to write to nodes. | Parameter | Type | Description | | - | - | - | | `external-id` | string | Enter the CDF view/container external ID. An error will occur if the specified view/container does not exist. | | `space` | string | Enter the CDF view/container space. An error will occur if the specified space does not exist. | | `version` | string | Enter the version of the view/container. It can be null if the view/container is not versioned. | #### `edges` Part of [`destination`](#queries.destination) configuration. The `edges` destination inserts the resulting data into edges containers. The edges destination is configured by setting the `type` parameter to `edges`. You have to add data-model. To ingest data into a edges, the query must produce columns named * `externalId` * `startNodeExternalId` * `startNodeSpace` * `endNodeExternalId` * `endNodeSpace` * `typeExternalId` * `typeSpace` | Parameter | Type | Description | | - | - | - | | `type` | always `edges` | Type of CDF destination, set to `edges` to write data to CDF edges. | | `space` | string | Enter the CDF space name. An error will occur if the specified space does not exist. | | [`view`](#queries.destination.edges.view) | object | Enter the CDF view/container configuration. This is required to write to edges. | ##### `view` Part of [`edges`](#queries.destination.edges) configuration. Enter the CDF view/container configuration. This is required to write to edges. | Parameter | Type | Description | | - | - | - | | `external-id` | string | Enter the CDF view/container external ID. An error will occur if the specified view/container does not exist. | | `space` | string | Enter the CDF view/container space. An error will occur if the specified space does not exist. | | `version` | string | Enter the version of the view/container. It can be null if the view/container is not versioned. | ### `schedule` Part of [`queries`](#queries) configuration. Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise. Either one of the following options: * [Fixed interval](#queries.schedule.fixed_interval) * [CRON expression](#queries.schedule.cron_expression) **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: interval expression: 1h ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: cron expression: 0 7-17 * * 1-5 ``` #### `fixed_interval` Part of [`schedule`](#queries.schedule) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `interval` | **Required.** Type of time interval configuration. Use `interval` to configure a fixed interval. | | `expression` | string | **Required.** Enter a time interval, with a unit. Avaiable units are `s` (seconds), `m` (minutes), `h` (hours) and `d` (days).

**Examples:**
`45s`
`15m`
`2h` | ####
`cron_expression` Part of [`schedule`](#queries.schedule) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `cron` | **Required.** Type of time interval configuration. Use `cron` to configure CRON schedule. | | `expression` | string | **Required.** Enter a CRON expression. See [crontab.guru](https://crontab.guru) for a guide on writing CRON expressions.

**Example:**
`*/15 8-16 * * *` | ##
`databases` Global parameter. List of databases to connect to Each element of this list should be a configuration for a database the extractor will connect to Either one of the following options: * [ODBC](#databases.odbc) * [PostgreSQL](#databases.postgresql) * [Oracle DB](#databases.oracle_db) * [Snowflake](#databases.snowflake) * [MongoDB](#databases.mongodb) * [Azure Cosmos DB](#databases.azure_cosmos_db) * [Local spreadsheet files](#databases.local_spreadsheet_files) * [Amazon Dynamo DB](#databases.amazon_dynamo_db) * [Amazon Redshift](#databases.amazon_redshift) * [Google BigQuery](#databases.google_bigquery) **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} databases: - type: odbc name: my-odbc-database connection-string: DRIVER={Oracle 19.3};DBQ=localhost:1521/XE;UID=SYSTEM;PWD=oracle - type: postgres name: postgres-db host: pg.company.com user: postgres password: secret123Pas$word - type: postgres name: postgres-db host: pg.company.com user: postgres password: secret123Pas$word source: name: test_source external-id: test_source_id_1 ``` ### `odbc` Part of [`databases`](#databases) configuration. Open Database Connectivity (ODBC) is a generic protocol for querying databases. To connect to a database using ODBC, you must first download and install an ODBC driver for your database system on the machine running the extractor. Consult the documentation or contact the vendor of your database system to find its driver. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: odbc name: asset-database connection-string: Driver={ODBC Driver 17 for SQL Server};Server=10.24.5.162;Database=assets;UID=extractorUser;PWD=myPassword; ``` | Parameter | Type | Description | | - | - | - | | `type` | always `odbc` | Select the type of database connection. Set to `odbc` for ODBC databases. | | `connection-string` | string | **Required.** Enter the ODBC connection string. This will differ between database vendors.

**Examples:**
`DRIVER={Oracle 19.3};DBQ=localhost:1521/XE;UID=SYSTEM;PWD=oracle`
`DSN={MyDatabaseDsn}` | | `response-encoding` | string | Override the encoding to expect on database responses if the driver does not adhere to the ODBC standard. Default is to follow the ODBC standard.

**Examples:**
`utf8`
`iso-8859-1` | | `query-encoding` | string | Override the encoding to use on database queries if the driver does not adhere to the ODBC standard. Default is to follow the ODBC standard.

**Examples:**
`utf8`
`iso-8859-1` | | `timeout` | integer | Enter the timeout in seconds for the ODBC connection and queries. The default value is no timeout.

Some ODBC drivers don't accept either the `SQL_ATTR_CONNECTION_TIMEOUT` or the `SQL_ATTR_QUERY_TIMEOUT` option. The extractor will log an exception with the message `Could not set timeout on the ODBC driver - timeouts might not work properly`. Extractions will continue regardless but without timeouts. To avoid this logline, you can disable timeouts for the database causing these problems. | | `batch-size` | integer | Enter the number of rows to fetch from the database at a time. Choose a value based on the load and capacity of the database, and fine-tune it for your environment. Default value is `1000`. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.odbc.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`odbc`](#databases.odbc) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `postgresql` Part of [`databases`](#databases) configuration. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: postgres name: my-database host: 10.42.39.12 user: extractor-user password: mySecretPassword ``` | Parameter | Type | Description | | - | - | - | | `type` | always `postgres` | **Required.** Type of database connection, set to `postgres` for PostgreSQL databases. | | `host` | string | **Required.** Enter the hostname or address of postgres database

**Examples:**
`123.234.123.234`
`postgres.my-domain.com`
`localhost` | | `user` | string | **Required.** Enter the username for postgres database | | `password` | string | **Required.** Enter the password for postgres database | | `database` | string | Enter the database name to use. The default is to use the user name. | | `port` | integer | Enter the port to connect to. Default value is `5432`. | | `timeout` | integer | Enter the timeout in seconds for the database connection and queries. The default value is no timeout. | | `batch-size` | integer | Enter the number of rows to fetch from the database at a time. Choose a value based on the load and capacity of the database, and fine-tune it for your environment. Default value is `1000`. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.postgresql.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`postgresql`](#databases.postgresql) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `oracle_db` Part of [`databases`](#databases) configuration. The Cognite DB Extractor can connect directly to an Oracle Database version 12.1 or later. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: oracle name: my-database host: 10.42.39.12 user: extractor-user password: mySecretPassword ``` | Parameter | Type | Description | | - | - | - | | `type` | always `oracle` | Type of database connection, set to `oracle` for Oracle databases. | | `host` | string | **Required.** Enter the hostname or address of oracle database

**Examples:**
`123.234.123.234`
`database.my-domain.com`
`localhost` | | `user` | string | **Required.** Enter the user name | | `password` | string | **Required.** Enter the user password | | `port` | integer | Enter the port to connect to. Default value is `1521`. | | `service-name` | string | Optionally specify the service name of the database to connect to | | `timeout` | integer | Timeout for statements to the database | | `batch-size` | integer | Enter the number of rows to fetch from the database at a time. Choose a value based on the load and capacity of the database, and fine-tune it for your environment. Default value is `1000`. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.oracle_db.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`oracle_db`](#databases.oracle_db) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `snowflake` Part of [`databases`](#databases) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `snowflake` | Type of database connection, set to `snowflake` for Snowflake data warehouses. | | `authentication_type` | string | Authentication type for Snowflake. Can be either `plain` or `oauth`. | | `user` | string | User name for Snowflake. `Required` if in case you are using Plain Authentication. | | `password` | string | Password for Snowflake. `Required` if in case you are using Plain Authentication. | | [`private_key`](#databases.snowflake.private_key) | object | Private key for Snowflake. Include to make use of private-key Authentication. | | `client_id` | string | Client ID for OAuth authentication. `Required` if in case you are using OAuth Authentication. | | `client_secret` | string | Client Secret for OAuth authentication. `Required` if in case you are using OAuth Authentication. | | `access_token_generate_url` | string | Azure OAuth 2.0 token endpoint. `Required` if in case you are using OAuth Authentication. | | `oauth_scopes` | string | Scopes for OAuth User. `Required` if in case you are using OAuth Authentication. | | `account` | string | **Required.** Snowflake account ID | | `compute_warehouse` | string | Snowflake warehouse to use. `Required` if in case you are using OAuth Authentication, Optional if in case you are using Plain Authentication. | | `organization` | string | **Required.** Snowflake organzation name | | `database` | string | **Required.** Snowflake database to use | | `schema` | string | **Required.** Snowflake schema to use | | `batch-size` | integer | Enter the number of rows to fetch from the database at a time. Choose a value based on the load and capacity of the database, and fine-tune it for your environment. Default value is `1000`. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.snowflake.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`private_key` Part of [`snowflake`](#databases.snowflake) configuration. Private key for Snowflake. Include to make use of private-key Authentication. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Path to the private key PEM file for Snowflake. | | `passphrase` | string | Passphrase for the private key file. Required if the private key is encrypted. | #### `source` Part of [`snowflake`](#databases.snowflake) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `mongodb` Part of [`databases`](#databases) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `mongodb` | Type of database connection, set to `mongodb` for MongoDB databases. | | `uri` | string | **Required.** Adress and authentication data for the database as a Uniform Resource Identifier (URI). You can read more about MongoDB URIs [here](https://www.mongodb.com/docs/manual/reference/connection-string).

**Example:**
`mongodb://mymongo:port/?retryWrites=true&connectTimeoutMS=10000` | | `database` | string | **Required.** Name of the related MongoDB database to use. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.mongodb.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`mongodb`](#databases.mongodb) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `azure_cosmos_db` Part of [`databases`](#databases) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `cosmosdb` | Type of database connection, set to `cosmosdb` for Cosmos DB databases. | | `host` | string | **Required.** Host address for the database

**Example:**
`https://my-cosmos-db.documents.azure.com` | | `key` | string | **Required.** Azure Key used to connect to the Cosms DB instance | | `database` | string | **Required.** Database name to use | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.azure_cosmos_db.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`azure_cosmos_db`](#databases.azure_cosmos_db) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `local_spreadsheet_files` Part of [`databases`](#databases) configuration. The Cognite DB extractor can run against excel spreadsheets and other files containting tabular data. The currently supported file types are * xlsx, xlsm and xlsb (modern Excel files) * xls (legacy excel files) * odf, ods and odt (OpenDocument Format, used by e.g. Libre Office and Open Office) * csv (Comma separated values) When using Excel or OpenDocument Format spreadsheets, you need to provide an additional `sheet` parameter in the associated [query configuration](#-queries). | Parameter | Type | Description | | - | - | - | | `type` | always `spreadsheet` | Type of connection, set to `spreadsheet` for local spreadsheet files. | | `path` | string | **Required.** Path to a single spreadsheet file

**Examples:**
`/path/to/my/excel/file.xlsx`
`./relative/path/file.csv`
`C:\\Users\\Robert\\Documents\\spreadsheet.xls` | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.local_spreadsheet_files.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`local_spreadsheet_files`](#databases.local_spreadsheet_files) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `amazon_dynamo_db` Part of [`databases`](#databases) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `dynamodb` | Type of database connection, set to `dynamodb` for DynamoDB databases. | | `aws-access-key-id` | string | **Required.** AWS authentication access key ID | | `aws-secret-access-key` | string | **Required.** AWS authentication access key secret | | `region-name` | string | **Required.** AWS region where your database is located.

**Example:**
`us-east-1` | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.amazon_dynamo_db.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`amazon_dynamo_db`](#databases.amazon_dynamo_db) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `amazon_redshift` Part of [`databases`](#databases) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `redshift` | Type of database connection, set to `redshift` for Redshift databases. | | `aws-access-key-id` | string | **Required.** AWS authentication access key ID | | `aws-secret-access-key` | string | **Required.** AWS authentication access key secret | | `region-name` | string | **Required.** AWS region where your database is located.

**Example:**
`us-east-1` | | `database` | string | **Required.** Redshift database | | `secret-arn` | string | AWS Secret ARN | | `cluster-identifier` | string | Name of the Redshift cluster to connect. This parameter is required when connecting to a managed Redshift cluster. | | `workgroup-name` | string | Name of the Redshift workgroup to connect. This parameter is mandatory when connecting to a Redshift Serverless database. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.amazon_redshift.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`amazon_redshift`](#databases.amazon_redshift) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ### `google_bigquery` Part of [`databases`](#databases) configuration. The Cognite DB Extractor can run against Google BigQuery using Google SQL(like) query. Because this extends the Google SDK, you also authenticate with the Google suggested authentication methods by setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your authentication key | Parameter | Type | Description | | - | - | - | | `type` | always `bigquery` | Type of database connection, set to `bigquery` for Google BigQuery | | `batch-size` | integer | Enter the number of rows to fetch from the database at a time. Choose a value based on the load and capacity of the database, and fine-tune it for your environment. Default value is `10000`. | | `name` | string | Enter a name for the database that will be used throughout the `queries` section and for logging. The name must be unique for each database in the configuration file. | | `timezone` | configuration for either or offset from UTC | Specify how the extractor should handle timestamps from the source when timezone data is absent. Either `local` for the local timezone on the machine the extractor is running on, `utc` for UTC, or a number for a numerical offset from UTC. Default value is `local`.

**Examples:**
`utc`
`-8`
`5.5` | | [`source`](#databases.google_bigquery.source) | object | Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | ####
`source` Part of [`google_bigquery`](#databases.google_bigquery) configuration. Represents the system from which the data originates, including identifying information such as the name and a unique external identifier. | Parameter | Type | Description | | - | - | - | | `name` | string | User given name of the source system or application from which the data originates | | `external-id` | string | A user given unique identifier (external-id) for the source. | ## `extractor` Global parameter. General extractor configuration | Parameter | Type | Description | | - | - | - | | [`state-store`](#extractor.state-store) | object | Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | | `upload-queue-size` | integer | Maximum size of upload queue. Upload to CDF will be triggered once this limit is reached. Default value is `100000`. | | `parallelism` | integer | Maximum number of parallel queries. Default value is `4`. | | `mode` | either `continuous` or `single` | Extractor mode. Continuous runs the configured queries using the schedules configured per query. Single runs the queries once each. | ### `state-store` Part of [`extractor`](#extractor) configuration. Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | Parameter | Type | Description | | - | - | - | | [`raw`](#extractor.state-store.raw) | object | A RAW state store stores the extraction state in a table in CDF RAW. | | [`local`](#extractor.state-store.local) | object | A local state store stores the extraction state in a JSON file on the local machine. | #### `raw` Part of [`state-store`](#extractor.state-store) configuration. A RAW state store stores the extraction state in a table in CDF RAW. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Enter the database name in CDF RAW. | | `table` | string | **Required.** Enter the table name in CDF RAW. | | `upload-interval` | integer | Enter the interval in seconds between each upload to CDF RAW. Default value is `30`. | #### `local` Part of [`state-store`](#extractor.state-store) configuration. A local state store stores the extraction state in a JSON file on the local machine. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Insert the file path to a JSON file. | | `save-interval` | integer | Enter the interval in seconds between each save. Default value is `30`. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/configuration/file-extractor Configuration reference for the Cognite File extractor with all available parameters and options. To configure the File extractor, you must create a configuration file. The file must be in [YAML](https://yaml.org) format. You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ## Using values from environment variables The configuration file allows substitutions with environment variables. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite: secret: ${COGNITE_CLIENT_SECRET} ``` will load the value from the `COGNITE_CLIENT_SECRET` environment variable into the `cognite/secret` parameter. You can also do string interpolation with environment variables, for example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN} ``` Implicit substitutions only work for unquoted value strings. For quoted strings, use the `!env` tag to activate environment substitution: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: !env 'http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}' ``` ## Using values from Azure Key Vault The DB extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `azure-keyvault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` Base configuration object | Parameter | Type | Description | | - | - | - | | `version` | either string or integer | Configuration file version | | `type` | either `local` or `remote` | Configuration file type. Either `local`, meaning the full config is loaded from this file, or `remote`, which means that only the `cognite` section is loaded from this file, and the rest is loaded from extraction pipelines. Default value is `local`. | | `cognite` | object | The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | | `logger` | object | The optional `logger` section sets up logging to a console and files. | | `files` | object | Configure files to be extracted to CDF. | | `extractor` | object | General configuration for the file extractor. | | `metrics` | object | The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | ## `cognite` Global parameter. The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | Parameter | Type | Description | | - | - | - | | `project` | string | Insert the CDF project name. | | `idp-authentication` | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | | `data-set` | object | Enter a data set the extractor should write data into | | `extraction-pipeline` | object | Enter the extraction pipeline used for remote config and reporting statuses | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | `timeout` | integer | Enter the timeout on requests to CDF, in seconds. Default value is `30`. | | `external-id-prefix` | string | Prefix on external ID used when creating CDF resources | | `connection` | object | Configure network connection details | ### `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | Parameter | Type | Description | | - | - | - | | `authority` | string | Insert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client ID from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | list | Enter a list of scopes requested for the token | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | `certificate` | object | Authenticate with a client certificate | #### `scopes` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Enter a list of scopes requested for the token Each element of this list should be a string. #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `data-set` Part of [`cognite`](#cognite) configuration. Enter a data set the extractor should write data into | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal ID | | `external-id` | string | Resource external ID | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Enter the extraction pipeline used for remote config and reporting statuses | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal ID | | `external-id` | string | Resource external ID | ### `connection` Part of [`cognite`](#cognite) configuration. Configure network connection details | Parameter | Type | Description | | - | - | - | | `disable-gzip` | boolean | Whether or not to disable gzipping of json bodies. | | `status-forcelist` | string | HTTP status codes to retry. Defaults to 429, 502, 503 and 504 | | `max-retries` | integer | Max number of retries on a given http request. Default value is `10`. | | `max-retries-connect` | integer | Max number of retries on connection errors. Default value is `3`. | | `max-retry-backoff` | integer | Retry strategy employs exponential backoff. This parameter sets a max on the amount of backoff after any request failure. Default value is `30`. | | `max-connection-pool-size` | integer | The maximum number of connections which will be kept in the SDKs connection pool. Default value is `50`. | | `disable-ssl` | boolean | Whether or not to disable SSL verification. | | `proxies` | object | Dictionary mapping from protocol to url. | #### `proxies` Part of [`connection`](#cognite.connection) configuration. Dictionary mapping from protocol to url. ## `logger` Global parameter. The optional `logger` section sets up logging to a console and files. | Parameter | Type | Description | | - | - | - | | `console` | object | Include the console section to enable logging to a standard output, such as a terminal window. | | `file` | object | Include the file section to enable logging to a file. The files are rotated daily. | | `metrics` | boolean | Enables metrics on the number of log messages recorded per logger and level. This requires `metrics` to be configured as well | ### `console` Part of [`logger`](#logger) configuration. Include the console section to enable logging to a standard output, such as a terminal window. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for console logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | ### `file` Part of `logger` configuration. Include the file section to enable logging to a file. The files are rotated daily. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for file logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | | `path` | string | **Required.** Insert the path to the log file. | | `retention` | integer | Specify the number of days to keep logs for. Default value is `7`. | ## `files` Global parameter. Configure files to be extracted to CDF. | Parameter | Type | Description | | - | - | - | | `file-provider` | configuration for either Local Files, SharePoint Online, FTP/FTPS, SFTP, GCP Cloud Storage, Azure Blob Storage, Azure Data Lake Storage, Samba, AWS S3 or Documentum | Configure a file provider for where the files are extracted from. | | `extensions` | list | List of file extensions to include. If left out, all file extensions will be allowed. | | `labels` | list | List of label external IDs to add to extracted files. | | `security-categories` | list | List of security category IDs to add to extracted files. | | `max-file-size` | either string or number | Maximum file size of files to include. Set to -1 to allow any file size. Syntax is `N(KB\|MB\|GB\|TB\|KiB\|miB\|GiB\|TiB)`. Note that the extractor supports files up to 1000GiB. Default value is `100GiB`. | | `with-metadata` | boolean | Add metadata extracted from the file source to files in CDF. | | `directory-prefix` | string | Prefix to add to all extracted file directories. | | `metadata-to-raw` | object | If this is configured, write metadata to a table in CDF RAW instead of files. | | `data_model` | object | When this is provided, all files metadata are uploaded to data models, hence metadata-to-raw becomes redundant. | | `destination_mode` | either `cdm` or `classic` | Mode of the file extractor. Can be 'cdm' for data models or 'classic' for Files. | | `destination_mode` | either `cdm` or `classic` | Mode of the file extractor. Can be 'cdm' for data models or 'classic' for Files. | | `source` | object | Sets the `Source` metadata field for the related files. When data modelling is set, it updates the underlying CogniteSourceSystem with the correspondent source. This is an optional parameter. | | `filter` | configuration for either And, Or, Not, Equals or In | | | `delete-behavior` | object | When files are no longer at the source: `soft` (mark with metadata) or `hard` (delete from CDF). | | `missing-as-deleted` | boolean | When set to `true`, treat files missing from the current source listing as logically deleted. This only affects files tracked in the [state store](#extractor.state-store). | The `missing-as-deleted` and `delete-behavior` parameters require a configured [state store](#extractor.state-store) to track files between runs. Without a state store, the extractor will not delete or mark files as deleted. ### `file-provider` Part of [`files`](#files) configuration. Configure a file provider for where the files are extracted from. Either one of the following options: * Local Files * SharePoint Online * FTP/FTPS * SFTP * GCP Cloud Storage * Azure Blob Storage * Azure Data Lake Storage * Samba * AWS S3 * Documentum #### `local_files` Part of `file-provider` configuration. Read files from a local folder. This file provider will recursively traverse the given path and extract all discovered files. **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: local path: /some/local/path ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: local path: - /some/local/path - /another/path ``` | Parameter | Type | Description | | - | - | - | | `type` | string | Select the type of file provider. Set to `local` for local files. | | `path` | configuration for either string or list | | | `use-relative-directory` | boolean | Set CDF metadata 'Directory' to the file relative path. When set to 'false', CDF directory will be set to the full folder path. Default value is `True`. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | ##### `ignore_patterns` Part of [`local_files`](#files.file-provider.local_files) configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of [`ignore_patterns`](#files.file-provider.local_files.ignore_patterns) configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `sharepoint_online` Part of [`file-provider`](#files.file-provider) configuration. Read files from one or more SharePoint Online sites. Multilevel nested subsites (two or more levels under a root site) require Cognite File extractor **2.12.0** or later and a [`structured`](#files.file-provider.sharepoint_online.paths.sharepoint_structured_path) path. Use a [`url`](#files.file-provider.sharepoint_online.paths.sharepoint_url_path) path only when the SharePoint location is at most one subsite deep. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: sharepoint_online client-id: ${SP_CLIENT_ID} client-secret: ${SP_CLIENT_SECRET} tenant-id: ${SP_AZURE_TENANT_ID} paths: - url: ${SP_EXTRACT_URL} ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: sharepoint_online client-id: ${SP_CLIENT_ID} client-secret: ${SP_CLIENT_SECRET} tenant-id: ${SP_AZURE_TENANT_ID} paths: - type: structured host: tenant.sharepoint.com site: RootSite sub_sites: Subsite1/Subsite2 library: Shared Documents folders: - Folder1 - Folder2/file.pdf ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `sharepoint_online` for SharePoint Online. | | `client-id` | string | **Required.** Enter the App registration client ID. | | `client-secret` | string | Enter the App registration secret. | | `certificate-path` | string | Enter the path to a certificate used for authentication. Either this, `client-secret`, or `certificate-data` must be specified. | | `certificate-data` | string | Provide authentication certificate data directly. | | `tenant-id` | string | **Required.** Enter the Azure tenant containing the App registration. | | `paths` | list | **Required.** Enter a list of SharePoint paths to extract from. Each item is either a full SharePoint URL ([`url`](#files.file-provider.sharepoint_online.paths.sharepoint_url_path)) or a structured path ([`structured`](#files.file-provider.sharepoint_online.paths.sharepoint_structured_path)) for two or more nested subsites. | | `resync-on-expired-delta-url` | boolean | If set to `true`, when SharePoint returns a 410 Gone error because the delta URL has expired, the extractor performs a full re-synchronization of the document library. Default value is `False`. | | `datetime-format` | string | Format string for timestamp metadata. Default value is `%Y-%m-%dT%H:%M:%SZ`. | | `extract-columns` | object | Extract SharePoint columns as metadata. This is a map from column names in SharePoint to the name you want the extracted columns to have in file metadata in CDF.

**Example:**
`{'columnNameInSharepoint': 'metadataNameInCdf'}` | | `restrict-to` | object | Restrict to extract only files visible to a given SharePoint Group or SiteUser.
Important: To use this, the extractor MUST authenticate to SharePoint Online using a certificate, NOT a client secret. | | `performance` | object | Configuration to tune the parallel calls made to the SharePoint MS Graph API. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | #####
`paths` Part of `sharepoint_online` configuration. Enter a list of SharePoint paths to extract from. Each item is either a full SharePoint URL for locations at most one subsite deep, or a structured path for two or more nested subsites. Either one of the following options: * [SharePoint URL path](#files.file-provider.sharepoint_online.paths.sharepoint_url_path) * [SharePoint Structured path](#files.file-provider.sharepoint_online.paths.sharepoint_structured_path) ###### `sharepoint_url_path` Part of [`paths`](#files.file-provider.sharepoint_online.paths) configuration. A SharePoint path specified as a URL. | Parameter | Type | Description | | - | - | - | | `type` | string | Set to `url` for a URL-based path. Default value is `url`. | | `url` | string | **Required.** URL to the SharePoint location you want to extract from. This can be the URL to a site, a document library, or a file or folder inside the library. Does not support paths with two or more nested subsites. For deeper hierarchies, use a [structured path](#files.file-provider.sharepoint_online.paths.sharepoint_structured_path). | | `recursive` | boolean | Whether to traverse into subfolders within the resolved document library for this path. Does not traverse into additional SharePoint subsites. Default value is `True`. | URL paths support SharePoint locations that are at most one subsite deep (for example, `https://tenant.sharepoint.com/sites/MySite/Subsite1/Shared%20Documents`). For two or more nested subsites, use a structured path. ###### `sharepoint_structured_path` Part of [`paths`](#files.file-provider.sharepoint_online.paths) configuration. A SharePoint path specified by its structural components (`host`, `site`, `sub_sites`, `library`, `folders`). Use this path type when you extract from two or more nested subsites under a root site collection. Structured paths support classic SharePoint site collections under `/sites/` only. They do not support `/teams/` URLs. Requires Cognite File extractor 2.12.0 or later. | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Set to `structured` for a structured path. | | `host` | string | **Required.** SharePoint host (e.g. `tenant.sharepoint.com`). | | `site` | string | **Required.** Root site collection name (the segment under `/sites/` for standard team sites). | | `sub_sites` | string | Optional. Nested subsite path under the root site, separated by `/` (for example, `Subsite1/Subsite2`). There is no depth limit. | | `library` | string | Optional. Document library (drive) name. If omitted and `recursive` is `true`, the extractor extracts files from all document libraries in the resolved subsite. If omitted and `recursive` is `false`, the extractor returns an error. | | `folders` | list | Optional. List of folders or files with folder paths (relative to the library root) to extract. | | `recursive` | boolean | Whether to traverse into subfolders within the resolved document library for this path. Does not traverse into additional SharePoint subsites. Default value is `True`. | ##### `extract-columns` Part of [`sharepoint_online`](#files.file-provider.sharepoint_online) configuration. Extract SharePoint columns as metadata. This is a map from column names in SharePoint to the name you want the extracted columns to have in file metadata in CDF. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} columnNameInSharepoint: metadataNameInCdf ``` | Parameter | Type | Description | | - | - | - | | Any string | string | Name of metadata field in CDF. | ##### `restrict-to` Part of [`sharepoint_online`](#files.file-provider.sharepoint_online) configuration. Restrict to extract only files visible to a given SharePoint Group or SiteUser. Important: To use this, the extractor MUST authenticate to SharePoint Online using a certificate, NOT a client secret. | Parameter | Type | Description | | - | - | - | | `group-id` | string | The ID of a SharePoint Group | | `username` | string | The "login name" of a SiteUser. This can be useful if you for example want to restrict extraction to the built-in "all users" SiteUser. | ##### `performance` Part of [`sharepoint_online`](#files.file-provider.sharepoint_online) configuration. Configuration to tune the parallel calls made to the SharePoint MS Graph API. | Parameter | Type | Description | | - | - | - | | `workers` | integer | Number of parallel workers used to read from SharePoint. Default value is `10`. | | `document-buffer` | integer | Number of document metadata instances to buffer before uploading the file contents to CDF. Default value is `60`. | ##### `ignore_patterns` Part of [`sharepoint_online`](#files.file-provider.sharepoint_online) configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of [`ignore_patterns`](#files.file-provider.sharepoint_online.ignore_patterns) configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `ftp/ftps` Part of [`file-provider`](#files.file-provider) configuration. Read files from an FTP server. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: ftp host: ftp.myserver.com username: username password: ${FTP_PASSWORD} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `ftp` for FTP. | | `host` | string | **Required.** Host name for the FTP server.

**Example:**
`ftp.myserver.com` | | `port` | integer | FTP server port. Default value is `21`. | | `username` | string | **Required.** Username to use to login to the FTP server. | | `password` | string | **Required.** Password to use to login to the FTP server. | | `root-directory` | string | Root folder for extraction. Default value is `/`. | | `recursive` | boolean | Whether to recursively traverse sub-folders for files. Default value is `True`. | | `use-ssl` | boolean | Whether to connect using FTPS (FTP with SSL/TLS). Using SSL is strongly recommended. | | `certificate-file-path` | string | Path to SSL certificate authority certificate for the FTP server, useful if using self-signed certificates. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | #####
`ignore_patterns` Part of [`ftp/ftps`](#files.file-provider.ftp/ftps) configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `sftp` Part of [`file-provider`](#files.file-provider) configuration. Read files from an SFTP server, file transfer over SSH. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: sftp host: ftp.myserver.com username: username password: ${FTP_PASSWORD} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `sftp` for SFTP. | | `host` | string | **Required.** Host name for the SSH server.

**Example:**
`ftp.myserver.com` | | `port` | integer | SSH server port. Default value is `22`. | | `username` | string | **Required.** Username to use to login to the SSH server. | | `password` | string | Password to use to login to the SSH server. Either `password` or `key-path` is required. | | `key-path` | string | Path to SSH private key for the connection. Either `password` or `key-path` is required. | | `key-password` | string | Password for SSH private key if the key is encrypted. Only used in combination with `key-path`. | | `root-directory` | string | Root folder for extraction. Default value is `/`. | | `recursive` | boolean | Whether to recursively traverse sub-folders for files. Default value is `True`. | | [`ignore_patterns`](#files.file-provider.sftp.ignore_patterns) | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | #####
`ignore_patterns` Part of `sftp` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * [String](#files.file-provider.sftp.ignore_patterns.string) * [Pattern with flags](#files.file-provider.sftp.ignore_patterns.pattern_with_flags) ###### `string` Part of [`ignore_patterns`](#files.file-provider.sftp.ignore_patterns) configuration. Any file path that matches this string pattern will be ignored. ###### `pattern_with_flags` Part of [`ignore_patterns`](#files.file-provider.sftp.ignore_patterns) configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `gcp_cloud_storage` Part of `file-provider` configuration. Read files from a GCP Cloud Storage bucket. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: gcp_cloud_storage google-application-credentials: ${GOOGLE_APPLICATION_CREDENTIALS} bucket: bucket_name folders: - list - of - folders ``` | Parameter | Type | Description | | - | - | - | | `type` | string | Select the type of file provider. Set to `gcp_cloud_storage` for GCP Cloud Storage. | | `google-application-credentials` | string | Base-64 encoded GCP service account credentials. | | `bucket` | string | Name of GCP Cloud Storage bucket to fetch files from. | | `folders` | list | List of folders in bucket to fetch files from. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | ##### `folders` Part of `gcp_cloud_storage` configuration. List of folders in bucket to fetch files from. Each element of this list should be a string. ##### `ignore_patterns` Part of `gcp_cloud_storage` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `azure_blob_storage` Part of `file-provider` configuration. Read files from an Azure Blob Store. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: azure_blob_storage connection-string: ${AZURE_BLOB_STORAGE_CONNECTION_STRING} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `azure_blob_storage` for Azure Blob Storage. | | `connection-string` | string | **Required.** Azure Blob Storage connection string. | | `containers` | list | Optional list of containers to extract from. If left out or empty, all files will be read. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | ##### `containers` Part of `azure_blob_storage` configuration. Optional list of containers to extract from. If left out or empty, all files will be read. Each element of this list should be a string. ##### `ignore_patterns` Part of `azure_blob_storage` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `azure_data_lake_storage` Part of `file-provider` configuration. Read files from an Azure Data Lake Storage. **Example:** To authenticate using a connection-string: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: azure_data_lake authentication: type: connection-string connection-string: ${AZURE_DATA_LAKE_CONNECTION_STRING} ``` To authenticate using client credentials: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: azure_data_lake authentication: type: client-credentials account-url: ${AZURE_STORAGE_ACCOUNT_URL} tenant-id: ${AZURE_TENANT_ID} client-id: ${AZURE_CLIENT_ID} client-secret: ${AZURE_CLIENT_SECRET} ``` To authenticate using default credentials: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: azure_data_lake authentication: type: default account-url: ${AZURE_STORAGE_ACCOUNT_URL} managed-identity-client-id: ${AZURE_MANAGED_IDENTITY_CLIENT_ID} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `azure_data_lake` for Azure Blob Storage. | | `authentication` | object | **Required.** Configuration for authenticating to Azure Data Lake Storage. | | `file-systems` | list | Optional list of file systems to extract from. If left out or empty, all files will be read. | | `ignore_patterns` | string or object | File paths matching this pattern are ignored. Accepts a string or a pattern object with flags. | ##### `authentication` Part of `azure_data_lake_storage` configuration. The `authentication` section is used to define how the extractor authenticates with Azure Data Lake Storage. | Parameter | Type | Description | | - | - | - | | `type` | string | **Required** Authentication method: either `connection-string` or `client-credentials` or `default`. | | `connection-string` | string | **Required** when `type` is `connection-string`. Full Azure storage connection string. | | `account-url` | string | **Required** when `type` is `client-credentials`. or `default`. The account DFS/Blob endpoint (for example, the storage account URL). | | `tenant-id` | string | **Required** when `type` is `client-credentials`. Azure AD tenant ID for the service principal. | | `client-id` | string | **Required** when `type` is `client-credentials`. Service principal (app) client ID. | | `client-secret` | string | **Required** when `type` is `client-credentials`. Service principal client secret; use environment variables or key vault where possible. | | `managed-identity-client-id` | string | The client ID for an Azure user-managed identity. Defaults to the system-managed identity if not specified. | ##### `file-systems` Part of `azure_data_lake_storage` configuration. Optional list of file systems to extract from. If left out or empty, all files will be read. Each element of this list should be a string. ##### `ignore_patterns` Part of `azure_data_lake_storage` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `samba` Part of `file-provider` configuration. Read files from a Samba file share. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: smb server: serverhost share-path: \\serverhost\share_path username: username password: ${SMB_PASSWORD} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `smb` for Samba. | | `share-path` | string | **Required.** Share path, i.e. name of path shared.

**Example:**
`\\server\share_path` | | `server` | string | **Required.** The IP or hostname of the server to connect to | | `port` | integer | Port to connect to on the Samba server. Default value is `445`. | | `username` | string | Username for authentication. | | `password` | string | Password for authentication. | | `require-signing` | boolean | Whether signing is required on messages sent to and received from the Samba server. Default value is `True`. | | `domain-controller` | string | The domain controller hostname. When set the file provider will send a DFS referral request to this hostname to populate the domain cache used for DFS connections or when connecting to `SYSVOL` or `NETLOGON` | | `skip-dfs` | boolean | Whether to skip using any DFS referral checks and treat any path as a normal path. This is only useful if there are problems with the DFS resolver or you wish to avoid the extra round trip(s) the resolver requires. | | `auth-protocol` | either `negotiate`, `kerberos` or `ntlm` | The protocol to use for authentication. Default value is `negotiate`. | | `require-secure-negotiate` | boolean | Whether to verify the negotiated dialects and capabilities on the connection to a share to protect against man in the middle downgrade attacks. Default value is `True`. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | #####
`ignore_patterns` Part of `samba` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `aws_s3` Part of `file-provider` configuration. Read files from an AWS S3 cloud bucket. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: aws_s3 bucket: some_bucket aws-access-key-id: ${AWS_ACCESS_KEY_ID} aws-secret-access-key: ${AWS_SECRET_ACCESS_KEY} ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `aws_s3` for AWS S3. | | `bucket` | string | **Required.** AWS S3 cloud bucket to read files from. Valid formats are `bucket-name` or `bucket-name/subfolder1/subfolder2`. | | `aws-access-key-id` | string | AWS access key ID to use for authentication. If left out, use default authentication configured on the machine the extractor is running on. | | `aws-secret-access-key` | string | AWS secret access key to use for authentication. If left out, use default authentication configured on the machine the extractor is running on. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | ##### `ignore_patterns` Part of `aws_s3` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | #### `documentum` Part of `file-provider` configuration. Read files from an OpenText Documentum server. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: documentum base-url: https://my.documentum.server/dctm username: username password: ${DOCUMENTUM_PASSWORD} field-map: name: - file_name - file_id - full_file_path external-id: - i_chronicle_id - r_object_id repositories: - name: some_repo query: SELECT * FROM tech_document WHERE object_name LIKE 'PREFIX%' ORDER BY r_object_id ``` | Parameter | Type | Description | | - | - | - | | `type` | string | **Required.** Select the type of file provider. Set to `documentum` to read from a Documentum server. | | `base-url` | string | **Required.** URL of the documentum server to read from. | | `username` | string | **Required.** Documentum server username. | | `password` | string | **Required.** Documentum server password. | | `repositories` | list | **Required.** List of documentum repositories to read from, and the query to make towards each repository. | | `ssl-verify` | boolean | Enable SSL certificate verification.

Warning: Disabling SSL verification is a potential security risk. Do not use this option over the internet, only on local networks secured through other means. Default value is `True`. | | `get-all-renditions` | boolean | Get all renditions of each document, not just the primary. | | `external-id-separator` | string | Text to use as a separator between the different parts of the created external IDs. Default value is `.`. | | `include-extension-in-file-names` | boolean | Include the file extension in the name of the file in CDF. For example, upload it as `My File.pdf` instead of `My File` | | `timeout` | string | Timeout on queries to documentum. On the form `N(s\|m\|h\|d)`. Default value is `5m`. | | `field-map` | object | Optional. For each supported `field-map` parameter, list Documentum property names in priority order. The extractor uses the first property name in each list that exists on the document.

Omit the whole object to use [built-in defaults](#field-map-defaults). | | `performance` | object | Configuration to tune the parallelism of the file extractor. | | `ignore_patterns` | configuration for either String or Pattern with flags | Any file path that matches this pattern will be ignored. | #####
`repositories` Part of `documentum` configuration. List of documentum repositories to read from, and the query to make towards each repository. Each element of this list should be an a documentum repository the file extractor should read from, and the query it should use. | Parameter | Type | Description | | - | - | - | | `name` | string | **Required.** Name of the documentum repository. | | `query` | string | **Required.** Query to retrieve file info from the repository.

Warning: The query *must* return a consistent ordering. To ensure this, add an `ORDER BY` clause for some ID field.

**Example:**
`SELECT * FROM tech_document WHERE object_name LIKE 'PREFIX%' ORDER BY r_object_id` | #####
`field-map` Part of `documentum` configuration. For each supported `field-map` parameter, list Documentum **property names in priority order**. The extractor reads the document's properties and uses the **first property name in each list that exists on the document**. If none of those properties exist, it follows the behavior in the table. Each list item is a string (a property name). The [Documentum example](#files.file-provider.documentum) shows `field-map` with prioritized property names for the `name` and `external-id` parameters. **Default `field-map` when you omit the section:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} field-map: external-id: - i_chronicle_id name: - object_name file-extension: - dos_extension - format_name modify-date: - r_modify_date mime-type: - mime_type ``` | Parameter | Type | Description | | - | - | - | | `external-id` | list | Contributes to the file external ID in CDF, together with `external-id-separator` and the extension. If no listed property exists, extraction fails for that document. | | `name` | list | File name in CDF. If no listed property exists, the extractor logs a warning and continues. | | `file-extension` | list | File extension, for example `pdf`. If no listed property exists, the extractor uses `bin` and logs a warning. | | `modify-date` | list | Source modified time. If no listed property exists, the extractor logs a warning and continues. | | `mime-type` | list | MIME type. If no listed property exists, the type is inferred from the file extension. | The file external ID in CDF combines the Documentum value resolved for `external-id`, `external-id-separator`, and the resolved file extension. ##### `performance` Part of `documentum` configuration. Configuration to tune the parallelism of the file extractor. | Parameter | Type | Description | | - | - | - | | `workers` | integer | Number of parallel workers used to read from documentum. Default value is `30`. | | `document-buffer` | integer | Number of document metadata instances to buffer before uploading the file contents to CDF. Default value is `60`. | | `page-buffer` | integer | Number of document metadata pages to buffer. Default value is `5`. | ##### `ignore_patterns` Part of `documentum` configuration. Any file path that matches this pattern will be ignored. Either one of the following options: * String * Pattern with flags ###### `pattern_with_flags` Part of `ignore_patterns` configuration. | Parameter | Type | Description | | - | - | - | | `pattern` | string | Pattern string | | `flags` | either `a` or `i` | | ### `extensions` Part of `files` configuration. List of file extensions to include. If left out, all file extensions will be allowed. Each element of this list should be a string. ### `labels` Part of `files` configuration. List of label external IDs to add to extracted files. Each element of this list should be a string. ### `security-categories` Part of `files` configuration. List of security category IDs to add to extracted files. Each element of this list should be an integer. ### `metadata-to-raw` Part of `files` configuration. If this is configured, write metadata to a table in CDF RAW instead of files. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Write file metadata to this Raw database. | | `table` | string | **Required.** Write file metadata to this Raw table. | ### `data_model` Part of `files` configuration. When this is provided, all files metadata are uploaded to data models, hence metadata-to-raw becomes redundant. | Parameter | Type | Description | | - | - | - | | `space` | string | | ### `source` Part of `files` configuration. Sets the `Source` metadata field for the related files. When data modelling is set, it updates the underlying [CogniteSourceSystem](/cdf/dm/dm_reference/dm_core_data_model#cognitesourcesystem) with the correspondent source. This is an optional parameter. | Parameter | Type | Description | | - | - | - | | `name` | string | | | `external_id` | string | | ### `filter` Part of `files` configuration. Either one of the following options: * And * Or * Not * Equals * In **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} and: - equals: property: some_metadata_field value: some_metadata_value - not: in: property: some_metadata_field values: - metadata - values ``` #### `and` Part of `filter` configuration. Matches if all sub filters match. | Parameter | Type | Description | | - | - | - | | `and` | list | **Required.** List of sub filters, all of these must match. | ##### `and` Part of `and` configuration. List of sub filters, all of these must match. Each element of this list should be a configuration for either And, Or, Not, Equals or In. #### `or` Part of `filter` configuration. Matches if any of the sub filters match. | Parameter | Type | Description | | - | - | - | | `or` | list | **Required.** List of sub filters, at least one of these must match. | ##### `or` Part of `or` configuration. List of sub filters, at least one of these must match. Each element of this list should be a configuration for either And, Or, Not, Equals or In. #### `not` Part of `filter` configuration. Matches if the sub filter does not match. | Parameter | Type | Description | | - | - | - | | `not` | configuration for either And, Or, Not, Equals or In | | #### `equals` Part of `filter` configuration. Matches if the property on the file is equal to the given value. | Parameter | Type | Description | | - | - | - | | `equals` | object | **Required.** Equality filter. | ##### `equals` Part of `equals` configuration. Equality filter. | Parameter | Type | Description | | - | - | - | | `property` | string | **Required.** File property name. | | `value` | string | **Required.** Property value to match. | #### `in` Part of `filter` configuration. Matches if the property on the file is equal to one of the given values. | Parameter | Type | Description | | - | - | - | | `in` | object | **Required.** In filter. | ##### `in` Part of `in` configuration. In filter. | Parameter | Type | Description | | - | - | - | | `property` | string | **Required.** File property name. | | `values` | list | Property values. One of these must match. | ###### `values` Part of `in` configuration. Property values. One of these must match. Each element of this list should be a string. ### `delete-behavior` Part of `files` configuration. Configures how to handle files no longer present at the source when [missing-as-deleted](#files.missing-as-deleted) is enabled. | Parameter | Type | Description | | - | - | - | | `mode` | either `soft` or `hard` | `soft`: add a metadata field to the file in CDF (given by `key`). `hard`: delete the file from CDF. | | `key` | string | Metadata field to add to the deleted file in CDF when `mode` is `soft`. Default value is `deleted`. | The 'missing-as-deleted' and 'delete-behavior' parameters require a configured [state store](#extractor.state-store) to track files between runs. Without a state store, the extractor will not delete or mark files as deleted. ## `extractor` Global parameter. General configuration for the file extractor. | Parameter | Type | Description | | - | - | - | | `state-store` | object | Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | | `errors-threshold` | integer | Maximum number of retries for fallible operations in the extractor. Default value is `5`. | | `failed-report-path` | string | Path to the failure report file. The report is a .jsonl file that contains the list of failed files and details regarding the errors triggered during the extraction. | | `upload-queue-size` | integer | Maximum number of files in the upload queue at a time. Default value is `10`. | | `parallelism` | integer | Maximum number of files to upload to CDF in parallel. Note that these files are streamed directly from the source, so this is also the number of parallel downloads. Default value is `4`. | | `dry-run` | boolean | Run the extractor in dry run mode. If set to true, nothing will be uploaded to CDF and no states will be stored. This means that we will load file metadata from the source, but not download any files. | | `schedule` | configuration for either Cron Expression or Interval | File extractor schedule.

**Examples:**
`{'type': 'cron', 'expression': '*/30 * * * *'}`
`{'type': 'interval', 'expression': '10m'}` | ###
`state-store` Part of `extractor` configuration. Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. The state store is also required for deletion ([`missing-as-deleted`](#files.missing-as-deleted) and [`delete-behavior`](#files.delete-behavior)) to take effect. We support multiple state stores, but you can only configure one at a time. | Parameter | Type | Description | | - | - | - | | `raw` | object | A RAW state store stores the extraction state in a table in CDF RAW. | | `local` | object | A local state store stores the extraction state in a JSON file on the local machine. | #### `raw` Part of `state-store` configuration. A RAW state store stores the extraction state in a table in CDF RAW. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Enter the database name in CDF RAW. | | `table` | string | **Required.** Enter the table name in CDF RAW. | | `upload-interval` | integer | Enter the interval in seconds between each upload to CDF RAW. Default value is `30`. | #### `local` Part of `state-store` configuration. A local state store stores the extraction state in a JSON file on the local machine. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Insert the file path to a JSON file. | | `save-interval` | integer | Enter the interval in seconds between each save. Default value is `30`. | ### `schedule` Part of `extractor` configuration. File extractor schedule. Either one of the following options: * Cron Expression * Interval **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: cron expression: '*/30 * * * *' ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: interval expression: 10m ``` #### `cron_expression` Part of `schedule` configuration. | Parameter | Type | Description | | - | - | - | | `type` | string | | | `expression` | string | **Required.** Cron expression schedule.

**Example:**
`*/30 * * * *` | ####
`interval` Part of `schedule` configuration. | Parameter | Type | Description | | - | - | - | | `type` | string | | | `expression` | string | **Required.** Fixed time interval. On the form `N(s\|m\|h\|d)`.

**Examples:**
`10m`
`3h` | ##
`metrics` Global parameter. The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | Parameter | Type | Description | | - | - | - | | `push-gateways` | list | List of prometheus pushgateway configurations | | `cognite` | object | Push metrics to CDF timeseries. Requires CDF credentials to be configured | | `server` | object | The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | ### `push-gateways` Part of `metrics` configuration. List of prometheus pushgateway configurations Each element of this list should be a the push-gateways sections contain a list of metric destinations. | Parameter | Type | Description | | - | - | - | | `host` | string | Enter the address of the host to push metrics to. | | `job-name` | string | Enter the value of the `exported_job` label to associate metrics with. This separates several deployments on a single pushgateway, and should be unique. | | `username` | string | Enter the credentials for the pushgateway. | | `password` | string | Enter the credentials for the pushgateway. | | `clear-after` | either null or integer | Enter the number of seconds to wait before clearing the pushgateway. When this parameter is present, the extractor will stall after the run is complete before deleting all metrics from the pushgateway. The recommended value is at least twice that of the scrape interval on the pushgateway. This is to ensure that the last metrics are gathered before the deletion. Default is disabled. | | `push-interval` | integer | Enter the interval in seconds between each push. Default value is `30`. | ### `cognite` Part of `metrics` configuration. Push metrics to CDF timeseries. Requires CDF credentials to be configured | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | **Required.** Prefix on external ID used when creating CDF time series to store metrics. | | `asset-name` | string | Enter the name for a CDF asset that will have all the metrics time series attached to it. | | `asset-external-id` | string | Enter the external ID for a CDF asset that will have all the metrics time series attached to it. | | `push-interval` | integer | Enter the interval in seconds between each push to CDF. Default value is `30`. | | `data-set` | object | Data set the metrics will be created under | #### `data-set` Part of `cognite` configuration. Data set the metrics will be created under | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal ID | | `external-id` | string | Resource external ID | ### `server` Part of `metrics` configuration. The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | Parameter | Type | Description | | - | - | - | | `host` | string | Host to run the prometheus server on. Default value is `0.0.0.0`. | | `port` | integer | Local port to expose the prometheus server on. Default value is `9000`. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/configuration/piaf Complete reference for configuring the PI AF extractor including connection settings, extraction options, and destination configuration. To configure the PI AF extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. You must name the configuration file config.yml. You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ## Before you start * Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`. * The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway. * Set up an [extraction pipeline](/cdf/integration/guides/interfaces/add_integrations) and note the external ID. ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 pi: # Required, host for the PI server host: "${PI_HOST}" # Windows username on the PI server username: "${PI_USERNAME}" # Windows password on the PI server password: "${PI_PASSWORD}" destination: # Change these to make sure you get the data somewhere unique in Raw database: piaf elements-table: elements cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: "${COGNITE_PROJECT}" # This is for microsoft as IdP, to use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} logger: console: level: "information" file: level: "debug" path: "logs/log.txt" ``` ## Intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. ## Using values from Azure Key Vault The PI AF extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `azure-keyvault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the PI AF extractor PI AF Extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | [`pi`](#pi) | object | Include the `pi` section to configure the connection to the PI AF system. | | [`cognite`](#cognite) | object | Configure connection to CDF. | | [`extraction`](#extraction) | object | Include the `extraction` section to configure how to extract data from PI AF. | | [`destination`](#destination) | object | Include the `destination` section to configure the destination for the extracted data. Currently this is only the CDF staging area (RAW). | | [`metrics`](#metrics) | object | Configuration for publishing metrics. | | [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | ## `pi` Global parameter. Include the `pi` section to configure the connection to the PI AF system. This is how the PI AF extractor selects a system: * If you configure `system-name`, the extractor selects the system by name from the preconfigured list of PI system on the machine the extractor runs on. * If you configure `host`, the extractor selects a PI system running on the PI server's host. * If you don't configure either of these parameters, the extractor selects the default system on the machine the extractor runs on. If there is no default system, the extractor selects the first system from the preconfigured PI system list on the machine the extractor runs on. | Parameter | Type | Description | | - | - | - | | `host` | string | Insert the base URL of the PI server's host. If you don't enter any value, you must configure a PI system in the installed SDK on the machine the extractor runs on. | | `username` | string | **Required.** Insert the Windows username on the PI server. | | `password` | string | **Required.** Insert the Windows password on the PI server. | | `system-name` | string | Enter the name of the PI system you want to use. This is used instead of `host` to select a PI system. | | `database-name` | string | Enter the name of the PI database you want to use. The default value is the **default** database configured on the machine the extractor runs on or the first database in the list of no default database in configured. | ## `cognite` Global parameter. Configure connection to CDF. | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | ###
`idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ## `extraction` Global parameter. Include the `extraction` section to configure how to extract data from PI AF. | Parameter | Type | Description | | - | - | - | | [`elements`](#extraction.elements) | object | Configuration for extraction PI AF Elements | | `update-period` | string | Enter the time between each time the extractor reads update events from the PI AF server. This is used to partiall refresh the PI AF elements to get newly created elements, or any changes to attribute values. Format is as given in [Intervals](#intervals).

For instance, `2h` means incremental updates run every other hour, starting at extractor startup. The extractor won't read updates at all if you set this parameter to `0` or a negative value. If both this parameter and `refresh-period` are set to `0` or a negative value, the extractor quits after reading all elements or after hitting the limit set in `elements.limit`. Default value is `0s`. | | `keep-alive` | string | Time between each time the extractor checks for changes to system and database status. This serves as a kind of keep alive, which may be necessary in some cases where the connection is timed out by an external mechanism. Format is as given in [Intervals](#intervals).

If this is 0 or negative, the extractor will not make keep alive requests. Default value is `5m`. | | `refresh-period` | string | Time between each time the extractor performs a full refresh, reading all data from the PI AF server. Format is as given in [Intervals](#intervals).

If this is 0 or negative, the extractor will only read all data on startup. Default value is `0s`. | ###
`elements` Part of [`extraction`](#extraction) configuration. Configuration for extraction PI AF Elements | Parameter | Type | Description | | - | - | - | | `chunk` | integer | Insert the maximum number of PI AF elements to read **per request to PI**. These are immediately written to CDF RAW. Default value is `1000`. | | `limit` | integer | Insert the total maximum number of PI AF elements to read. Use this to get a reasonable subset of the server for testing. Note that this doesn't work if [`extraction.update-period`](#extraction) is configured. | | `query` | string | Insert the string query, see [Aveva documentation](https://docs.aveva.com/bundle/af-sdk/page/html/pipoint-query-syntax-overview.htm) | | `flatten-attributes` | boolean | True to flatten attributes into a separate Raw table. If `false`, all attributes belonging to an element will be extracted as fields on elements in the elements table | ## `destination` Global parameter. Include the `destination` section to configure the destination for the extracted data. Currently this is only the CDF staging area (RAW). | Parameter | Type | Description | | - | - | - | | `database` | string | Insert the CDF RAW database to extract data to. If no database exists, the extractor creates a database. Default value is `piaf`. | | `elements-table` | string | Enter the table name for the PI AF elements in the CDF RAW database. If no table exists, the database creates a table. Default value is `elements`. | | `unit-of-measure-classes-table` | string | Enter the table name for unit-of-measure classes in the CDF RAW database. If no table exists, the extractor creates a table. Default value is `unit-of-measure-classes`. | | `attributes-table` | string | Enter the table name for the attributes in the CDF RAW database to be used if [`elements.flatten-attributes`](#extraction.elements) is set to `true`. If no table exists, the extractor creates a table. Default value is `attributes`. | ## `metrics` Global parameter. Configuration for publishing metrics. | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. | | [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for having the extractor start a Prometheus scrape server on a local port. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.

**Examples:**
`localhost`
`0.0.0.0` | | `port` | integer | **Required.** The port used for a local Prometheus server. | ###
`push-gateways` Part of [`metrics`](#metrics) configuration. A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

**Example:**
`http://my.pushgateway:9091` | | `job` | string | **Required.** Name of the Prometheus pushgateway job. | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ##
`logger` Global parameter. Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | ### `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/configuration/sap Configure the SAP extractor with YAML files to connect to SAP instances and extract data to CDF. To configure the SAP extractor, you must create a configuration file. The file must be in [YAML](https://yaml.org) format. You can use the sample minimal configuration file included with the extractor packages as a starting point for your configuration settings. The configuration file contains the global parameter `version`, which holds the version of the configuration schema. This article describes version 1. You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ## Using values from environment variables The configuration file allows substitutions with environment variables. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite: secret: ${COGNITE_CLIENT_SECRET} ``` will load the value from the `COGNITE_CLIENT_SECRET` environment variable into the `cognite/secret` parameter. You can also do string interpolation with environment variables, for example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN} ``` Implicit substitutions only work for unquoted value strings. For quoted strings, use the `!env` tag to activate environment substitution: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: !env 'http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}' ``` ## Using values from Azure Key Vault The DB extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `azure-keyvault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` Base configuration object | Parameter | Type | Description | | - | - | - | | `version` | either string or integer | Configuration file version | | `type` | either `local` or `remote` | Configuration file type. Either `local`, meaning the full config is loaded from this file, or `remote`, which means that only the `cognite` section is loaded from this file, and the rest is loaded from extraction pipelines. Default value is `local`. | | [`cognite`](#cognite) | object | The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | | [`logger`](#logger) | object | The optional `logger` section sets up logging to a console and files. | | [`metrics`](#metrics) | object | The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | | [`sap`](#sap) | list | List of SAP instances to connect to | | [`endpoints`](#endpoints) | list | List of endpoints to query | | [`extractor`](#extractor) | object | General extractor configuration | ## `cognite` Global parameter. The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | Parameter | Type | Description | | - | - | - | | `project` | string | Insert the CDF project name. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | | [`data-set`](#cognite.data-set) | object | Enter a data set the extractor should write data into | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Enter the extraction pipeline used for remote config and reporting statuses | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | `timeout` | integer | Enter the timeout on requests to CDF, in seconds. Default value is `30`. | | `external-id-prefix` | string | Prefix on external ID used when creating CDF resources | | [`connection`](#cognite.connection) | object | Configure network connection details | ### `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | Parameter | Type | Description | | - | - | - | | `authority` | string | Insert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | [`scopes`](#cognite.idp-authentication.scopes) | list | Enter a list of scopes requested for the token | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `scopes` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Enter a list of scopes requested for the token Each element of this list should be a string. #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `data-set` Part of [`cognite`](#cognite) configuration. Enter a data set the extractor should write data into | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Enter the extraction pipeline used for remote config and reporting statuses | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `connection` Part of [`cognite`](#cognite) configuration. Configure network connection details | Parameter | Type | Description | | - | - | - | | `disable-gzip` | boolean | Whether or not to disable gzipping of json bodies. | | `status-forcelist` | string | HTTP status codes to retry. Defaults to 429, 502, 503 and 504 | | `max-retries` | integer | Max number of retries on a given http request. Default value is `10`. | | `max-retries-connect` | integer | Max number of retries on connection errors. Default value is `3`. | | `max-retry-backoff` | integer | Retry strategy employs exponential backoff. This parameter sets a max on the amount of backoff after any request failure. Default value is `30`. | | `max-connection-pool-size` | integer | The maximum number of connections which will be kept in the SDKs connection pool. Default value is `50`. | | `disable-ssl` | boolean | Whether or not to disable SSL verification. | | [`proxies`](#cognite.connection.proxies) | object | Dictionary mapping from protocol to url. | #### `proxies` Part of [`connection`](#cognite.connection) configuration. Dictionary mapping from protocol to url. ## `logger` Global parameter. The optional `logger` section sets up logging to a console and files. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Include the console section to enable logging to a standard output, such as a terminal window. | | [`file`](#logger.file) | object | Include the file section to enable logging to a file. The files are rotated daily. | | `metrics` | boolean | Enables metrics on the number of log messages recorded per logger and level. This requires `metrics` to be configured as well | ### `console` Part of [`logger`](#logger) configuration. Include the console section to enable logging to a standard output, such as a terminal window. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for console logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | ### `file` Part of [`logger`](#logger) configuration. Include the file section to enable logging to a file. The files are rotated daily. | Parameter | Type | Description | | - | - | - | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for file logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | | `path` | string | **Required.** Insert the path to the log file. | | `retention` | integer | Specify the number of days to keep logs for. Default value is `7`. | ## `metrics` Global parameter. The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | Parameter | Type | Description | | - | - | - | | [`push-gateways`](#metrics.push-gateways) | list | List of prometheus pushgateway configurations | | [`cognite`](#metrics.cognite) | object | Push metrics to CDF timeseries. Requires CDF credentials to be configured | | [`server`](#metrics.server) | object | The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | ### `push-gateways` Part of [`metrics`](#metrics) configuration. List of prometheus pushgateway configurations Each element of this list should be a the push-gateways sections contain a list of metric destinations. | Parameter | Type | Description | | - | - | - | | `host` | string | Enter the address of the host to push metrics to. | | `job-name` | string | Enter the value of the `exported_job` label to associate metrics with. This separates several deployments on a single pushgateway, and should be unique. | | `username` | string | Enter the credentials for the pushgateway. | | `password` | string | Enter the credentials for the pushgateway. | | `clear-after` | either null or integer | Enter the number of seconds to wait before clearing the pushgateway. When this parameter is present, the extractor will stall after the run is complete before deleting all metrics from the pushgateway. The recommended value is at least twice that of the scrape interval on the pushgateway. This is to ensure that the last metrics are gathered before the deletion. Default is disabled. | | `push-interval` | integer | Enter the interval in seconds between each push. Default value is `30`. | ### `cognite` Part of [`metrics`](#metrics) configuration. Push metrics to CDF timeseries. Requires CDF credentials to be configured | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | **Required.** Prefix on external ID used when creating CDF time series to store metrics. | | `asset-name` | string | Enter the name for a CDF asset that will have all the metrics time series attached to it. | | `asset-external-id` | string | Enter the external ID for a CDF asset that will have all the metrics time series attached to it. | | `push-interval` | integer | Enter the interval in seconds between each push to CDF. Default value is `30`. | | [`data-set`](#metrics.cognite.data-set) | object | Data set the metrics will be created under | #### `data-set` Part of [`cognite`](#metrics.cognite) configuration. Data set the metrics will be created under | Parameter | Type | Description | | - | - | - | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `server` Part of [`metrics`](#metrics) configuration. The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | Parameter | Type | Description | | - | - | - | | `host` | string | Host to run the prometheus server on. Default value is `0.0.0.0`. | | `port` | integer | Local port to expose the prometheus server on. Default value is `9000`. | ## `sap` Global parameter. List of SAP instances to connect to Each element of this list should be a configuration of an SAP source Either one of the following options: * [SAP NetWeaver gateway](#sap.sap_netweaver_gateway) * [SAP SOAP Source](#sap.sap_soap_source) ### `sap_netweaver_gateway` Part of [`sap`](#sap) configuration. The SAP NetWeaver Gateway lets clients connect using the Open Data Protocol (OData). **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: odata source-name: mys4hana gateway-url: https://mys4hana.com/sap/opu/odata/sap/ client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: odata4 source-name: mys4hana gateway-url: https://mys4hana.com/sap/opu/odata4/sap client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` | Parameter | Type | Description | | - | - | - | | `type` | always `odata` | **Required.** Type of SAP source connection, set to `odata` for SAP OData v2 sources. Set to `odata4` for SAP OData v4 sources. | | `source-name` | string | **Required.** Enter a name for the source that will be used throughout the `endpoints` section and for logging. The name must be unique for each source in the configuration file. | | `gateway-url` | string | **Required.** Insert the SAP NetWeaver Gateway URL | | `client` | string | **Required.** Insert the SAP client number | | `username` | string | **Required.** Enter the SAP username to connect to the SAP NetWeaver Gateway. | | `password` | string | **Required.** Enter the SAP password to connect to the SAP NetWeaver Gateway | | `disable-ssl` | boolean | Disable the SSL certificate verification towards the SAP destination. Default value is set to False. | | `connection_check_timeout` | integer | Enter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is `60`. | | `language` | string | Enter the sap-language URL parameter. The default value is EN | | [`certificates`](#sap.sap_netweaver_gateway.certificates) | object | Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | | [`proxy`](#sap.sap_netweaver_gateway.proxy) | object | HTTP and/or HTTPS proxies to be used when connecting to the SAP source system. | | `timezone` | either `local` or `utc` | Specify how the extractor should handle the source time zones. Default value is `local`. | ####
`certificates` Part of [`sap_netweaver_gateway`](#sap.sap_netweaver_gateway) configuration. Certificates needed for authentication towards SAP instance. There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | Parameter | Type | Description | | - | - | - | | `ca-cert` | string | **Required.** Enter the path to the CA certificate file. | | `public-key` | string | **Required.** Enter the path to the public key file. | | `private-key` | string | **Required.** Enter the path to the private key file. | #### `proxy` Part of [`sap_netweaver_gateway`](#sap.sap_netweaver_gateway) configuration. HTTP and/or HTTPS proxies to be used when connecting to the SAP source system. | Parameter | Type | Description | | - | - | - | | `http` | string | Enter the address of the HTTP proxy | | `https` | string | Enter the address of the HTTPS proxy. | ### `sap_soap_source` Part of [`sap`](#sap) configuration. The SAP extractor can connect to SAP instances using SOAP, such as for SAP ERP SOAMANAGER **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: soap source-name: soap-funcloc wsdl-url: https://myerp.com/sap/bc/srt/wsdl/sap/bc/srt/rfc/sap/test/100/test_funcloc/test_funcloc?sap-client=100 client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` | Parameter | Type | Description | | - | - | - | | `type` | always `soap` | **Required.** Type of SAP source connection, set to `soap` for SAP SOAP sources. | | `source-name` | string | **Required.** Enter a name for the source that will be used throughout the `endpoints` section and for logging. The name must be unique for each source in the configuration file. | | `wsdl-url` | string | **Required.** Insert the [SOAP WSDL URL](https://help.sap.com/doc/saphelp_nw75/7.5.5/en-US/48/d185ca45885c97e10000000a42189d/content.htm?no_cache=true) related to the SAP ABAP webservice | | `strict-parser` | boolean | Flag to control how the extractor behave when parsing SOAP responses. Default value is set to True. | | `disable-ssl` | boolean | Disable the SSL certificate verification towards the SAP destination. Default value is set to False. | | `client` | string | **Required.** Insert the SAP client number | | `username` | string | **Required.** Enter the username to connect to the SAP Webservice. | | `password` | string | **Required.** Enter the password to connect to the SAP Webservice | | `language` | string | Enter the sap-language URL parameter. The default value is EN | | [`certificates`](#sap.sap_soap_source.certificates) | object | Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | | `connection_check_timeout` | integer | Enter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is `60`. | | `session_timeout` | integer | Enter the timeout (in seconds) period the extractor should use when fetching WSDL definitions from the SOAP source. Default value is `300`. | | `timezone` | either `local` or `utc` | Specify how the extractor should handle the source time zones. Default value is `local`. | ####
`certificates` Part of [`sap_soap_source`](#sap.sap_soap_source) configuration. Certificates needed for authentication towards SAP instance. There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | Parameter | Type | Description | | - | - | - | | `ca-cert` | string | **Required.** Enter the path to the CA certificate file. | | `public-key` | string | **Required.** Enter the path to the public key file. | | `private-key` | string | **Required.** Enter the path to the private key file. | ## `endpoints` Global parameter. List of endpoints to query Each element of this list should be a description of an endpoint to extract data from on one of the configured SAP sources. | Parameter | Type | Description | | - | - | - | | `name` | string | **Required.** Enter a name of this SAP endpoint that will be used to for logging. The name must be unique for each query in the configuration file. | | `source-name` | string | **Required.** Enter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the [`sap`](#sap) section. | | `sap-service` | string | **Required.** Enter the name of the related SAP service. For `odata` endpoints, it's the SAP OData service. For `soap` endpoints, it's the service defined in the WSDL document. | | `sap-entity` | string | **Required.** Enter the name of the related SAP entity. For `odata` endpoints, it's the name of the OData entity. For `soap` endpoints, it's the name of the SOAP operator as defined in the WSDL | | [`destination`](#endpoints.destination) | configuration for either RAW, Events, Assets, Time Series or Files | **Required.** The destination of the data in CDF. | | `pagination-type` | either `client`, `server` or `no-pagination` | OData pagination type when running full load (non incremental) from SAP. Default value is `no-pagination`. | | [`sap-key`](#endpoints.sap-key) | list | Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. | | [`select`](#endpoints.select) | list | Enter a list of fields to be selected from the SAP entity, using OData \$select operation. This parameter is available only for SAP OData endpoints. | | [`related-entities`](#endpoints.related-entities) | object | Sub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources. | | [`attachments`](#endpoints.attachments) | object | Extraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations) | | `request` | string | Enter the request to be sent to the SAP. This is a required parameter for `soap` endpoints. See the [Requests](#requests) section for more details. | | `incremental-field` | string | Enter the name of the field to use as the reference for incremental runs. If you omit this field, the extractor performs a full data load on every run. The SAP OData endpoint must return records ordered by this field in ascending order. See [Load data incrementally](/cdf/integration/guides/extraction/sap/sap_setup#load-data-incrementally-odata-only) for prerequisites and configuration details. | | `incremental-field-format` | string | Enter the date format of the incremental field so the extractor can convert it properly when running incremental queries. Example: `%H:%M:%S` is the date format for the date value `12:00:00`. This is an optional parameter. | | `initial-start` | string | Enter the initial value to be used when running incremental requests. Mandatory parameter when running incremental loads in SOAP endpoints. | | [`schedule`](#endpoints.schedule) | configuration for either Fixed interval or CRON expression | Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise.

**Examples:**
`{'schedule': {'type': 'interval', 'expression': '1h'}}`
`{'schedule': {'type': 'cron', 'expression': '0 7-17 * * 1-5'}}` | | [`extract-schema`](#endpoints.extract-schema) | object | If included, the extractor will extract the SAP entity schema to CDF RAW | | `filter` | string | Enter the filter query string. The `$filter` system query option allows clients to filter a collection of resources from the target SAP OData endpoint. This is only relevant for `odata` sources. | | `soap-operation` | string | Enter the name of the SOAP operation defined in the WSDL configuration. For `soap` endpoints, this is a required parameter. | | `soap-port` | string | Enter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for `soap` endpoints. | ### Requests The `request` parameter is part of the [`endpoints`](#endpoints) configuration, and is required `soap` endpoints. #### SOAP requests SAP ABAP Webservices are SOAP/based, meaning the requests to the SAP server must be in a valid XML format. The SAP extractor expects this XML to be added as a string in the `request` parameter. This is an example of a valid XML request to a SAP ABAP Webservice generated from a SAP Function Module: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} request: | String 57 String 58 S String 60 Strin S String 63 Stri 1010 Str String 67 I 1010 1010 ``` ###
`destination` Part of [`endpoints`](#endpoints) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.destination.raw) * [Events](#endpoints.destination.events) * [Assets](#endpoints.destination.assets) * [Time Series](#endpoints.destination.time_series) * [Files](#endpoints.destination.files) #### `raw` Part of [`destination`](#endpoints.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | #### `events` Part of [`destination`](#endpoints.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | - | - | - | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ##### `field-mapping` Part of [`events`](#endpoints.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | #### `assets` Part of [`destination`](#endpoints.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ##### `field-mapping` Part of [`assets`](#endpoints.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | #### `time_series` Part of [`destination`](#endpoints.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ##### `field-mapping` Part of [`time_series`](#endpoints.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | #### `files` Part of [`destination`](#endpoints.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | - | - | - | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | ### `sap-key` Part of [`endpoints`](#endpoints) configuration. Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. Each element of this list should be a string. ### `select` Part of [`endpoints`](#endpoints) configuration. Enter a list of fields to be selected from the SAP entity, using OData \$select operation. This parameter is available only for SAP OData endpoints. Each element of this list should be a string. ### `related-entities` Part of [`endpoints`](#endpoints) configuration. Sub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources. | Parameter | Type | Description | | - | - | - | | `to_expand` | string | Name of the value to be used in the OData \$expand operation | | `sap-service` | string | Enter the name of the related SAP service. For `odata` endpoints, it's the SAP OData service. For `soap` endpoints, it's the SOAP service defined in the WSDL document | | `sap-entity` | string | Enter the name of the related SAP entity. For `odata` endpoints, it's the name of the OData entity. For `soap` endpoints, it's the name of the SOAP object as defined in the WSDL. | | `source-name` | string | Enter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the [`sap`](#sap) section. | | [`destination`](#endpoints.related-entities.destination) | configuration for either RAW, Events, Assets, Time Series or Files | The destination of the data in CDF. | | [`sap-key`](#endpoints.related-entities.sap-key) | list | Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. | | `request` | string | Enter the request to be sent to the SAP when using SOAP as a SAP source. This is a required parameter for `soap` endpoints. See the [Requests](#requests) section for more details. | | `parent-field` | string | Enter the name of the field from the parent entity to be used as input key when querying the related entity. This parameter is mandatory when using SOAP sources. | | `soap-operation` | string | Enter the name of the SOAP operation defined in the WSDL configuration. For `soap` endpoints, this is a required parameter. | | `soap-port` | string | Enter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for `soap` endpoints. | #### `destination` Part of [`related-entities`](#endpoints.related-entities) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.related-entities.destination.raw) * [Events](#endpoints.related-entities.destination.events) * [Assets](#endpoints.related-entities.destination.assets) * [Time Series](#endpoints.related-entities.destination.time_series) * [Files](#endpoints.related-entities.destination.files) ##### `raw` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ##### `events` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | - | - | - | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.related-entities.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ###### `field-mapping` Part of [`events`](#endpoints.related-entities.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | ##### `assets` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.related-entities.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ###### `field-mapping` Part of [`assets`](#endpoints.related-entities.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | ##### `time_series` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.related-entities.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ###### `field-mapping` Part of [`time_series`](#endpoints.related-entities.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | ##### `files` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | - | - | - | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | #### `sap-key` Part of [`related-entities`](#endpoints.related-entities) configuration. Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. Each element of this list should be a string. ### `attachments` Part of [`endpoints`](#endpoints) configuration. Extraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations) | Parameter | Type | Description | | - | - | - | | [`destination`](#endpoints.attachments.destination) | configuration for either RAW, Events, Assets, Time Series or Files | The destination of the data in CDF. | | `object-key` | string | SAP primary key of the entity related to the attachment. This is a mandatory parameter. | | `business-object-type` | string | SAP Business Object Type related to the entity related to the attachment. For example, `BUS2038` is the standard business object type for SAP Maintenance Notifications. This is a mandatory parameter. | #### `destination` Part of [`attachments`](#endpoints.attachments) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.attachments.destination.raw) * [Events](#endpoints.attachments.destination.events) * [Assets](#endpoints.attachments.destination.assets) * [Time Series](#endpoints.attachments.destination.time_series) * [Files](#endpoints.attachments.destination.files) ##### `raw` Part of [`destination`](#endpoints.attachments.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ##### `events` Part of [`destination`](#endpoints.attachments.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | - | - | - | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.attachments.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ###### `field-mapping` Part of [`events`](#endpoints.attachments.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | ##### `assets` Part of [`destination`](#endpoints.attachments.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.attachments.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ###### `field-mapping` Part of [`assets`](#endpoints.attachments.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | ##### `time_series` Part of [`destination`](#endpoints.attachments.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | - | - | - | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.attachments.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ###### `field-mapping` Part of [`time_series`](#endpoints.attachments.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | - | - | - | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | ##### `files` Part of [`destination`](#endpoints.attachments.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | - | - | - | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | ### `schedule` Part of [`endpoints`](#endpoints) configuration. Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise. Either one of the following options: * [Fixed interval](#endpoints.schedule.fixed_interval) * [CRON expression](#endpoints.schedule.cron_expression) **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: interval expression: 1h ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: cron expression: 0 7-17 * * 1-5 ``` #### `fixed_interval` Part of [`schedule`](#endpoints.schedule) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `interval` | **Required.** Type of time interval configuration. Use `interval` to configure a fixed interval. | | `expression` | string | **Required.** Enter a time interval, with a unit. Avaiable units are `s` (seconds), `m` (minutes), `h` (hours) and `d` (days).

**Examples:**
`45s`
`15m`
`2h` | ####
`cron_expression` Part of [`schedule`](#endpoints.schedule) configuration. | Parameter | Type | Description | | - | - | - | | `type` | always `cron` | **Required.** Type of time interval configuration. Use `cron` to configure CRON schedule. | | `expression` | string | **Required.** Enter a CRON expression. See [crontab.guru](https://crontab.guru) for a guide on writing CRON expressions.

**Example:**
`*/15 8-16 * * *` | ###
`extract-schema` Part of [`endpoints`](#endpoints) configuration. If included, the extractor will extract the SAP entity schema to CDF RAW | Parameter | Type | Description | | - | - | - | | `type` | always `raw` | Type of CDF destination, always set to `raw` | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ## `extractor` Global parameter. General extractor configuration | Parameter | Type | Description | | - | - | - | | `mode` | either `continuous` or `single` | Extractor mode. Continuous runs the configured queries using the schedules configured per query. Single runs the queries once each. Default value is `single`. | | `parallelism` | integer | Maximum number of parallel queries. Default value is `10`. | | `upload-queue-size` | integer | Maximum size of upload queue. Upload to CDF will be triggered once this limit is reached. Default value is `50000`. | | `delta-padding-minutes` | integer | Insert the number of minutes to overlap when doing incremental load. When querying SAP for new or updated records, the extractor will read data starting from the last time the extractor ran minus this padding. This is to ensure some that there are some overlap in the queries and therefore no gaps. Default value is `1`. | | `chunk-size` | integer | Number of records to fetch in a single request to SAP. Default value is `1000`. | | [`state-store`](#extractor.state-store) | object | Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | ### `state-store` Part of [`extractor`](#extractor) configuration. Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | Parameter | Type | Description | | - | - | - | | [`raw`](#extractor.state-store.raw) | object | A RAW state store stores the extraction state in a table in CDF RAW. | | [`local`](#extractor.state-store.local) | object | A local state store stores the extraction state in a JSON file on the local machine. | #### `raw` Part of [`state-store`](#extractor.state-store) configuration. A RAW state store stores the extraction state in a table in CDF RAW. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Enter the database name in CDF RAW. | | `table` | string | **Required.** Enter the table name in CDF RAW. | | `upload-interval` | integer | Enter the interval in seconds between each upload to CDF RAW. Default value is `30`. | #### `local` Part of [`state-store`](#extractor.state-store) configuration. A local state store stores the extraction state in a JSON file on the local machine. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Insert the file path to a JSON file. | | `save-interval` | integer | Enter the interval in seconds between each save. Default value is `30`. | # About the Cognite DB extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/db Connect to databases and load data into Cognite Data Fusion (CDF) with the general-purpose DB extractor. **Deploy with the Cognite Toolkit:** The [Foundation deployment pack](/cdf/deploy/cdf_toolkit/references/packages/foundation) includes a `cdf_db_extractor` module with ready-made extraction pipeline configs — use it instead of configuring the extractor pipeline by hand. Run the extractors once for single extraction or set up regular runs to keep up with additions to the database tables. You can [load data incrementally](/cdf/integration/guides/extraction/db/db_setup#load-data-incrementally) if you only want to process new or updated rows since the last extraction. The extractor is available as a Windows executable file or as a Docker container for Unix-like systems, such as Mac OS and Linux. # Metrics Source: https://docs.cognite.com/cdf/integration/guides/extraction/db/db_metrics Performance metrics available from the DB extractor for remote monitoring and debugging. ## Prometheus metrics This section contains the full list of available metrics from the extractor. For information on the different types of metrics, see the Prometheus documentation. In general, Counter starts at zero and increases during a run of the extractor, while Gauge may change freely. | Name | Type | Purpose | | - | - | - | | db\_extractor\_start\_time | Gauge | The timestamp (seconds) of when the extractor last started. | | db\_extractor\_finish\_time | Gauge | The timestamp (seconds) of when the extractor last finished. | | db\_extractor\_memory\_bytes | Gauge | The memory usage of extractor in bytes. | | db\_extractor\_cpu\_percent | | Total The CPU usage of the extractor (0 — 100). Notice that this value is summed across all threads, so the value might exceed 100. | | db\_extractor\_num\_threads | Gauge | The number of threads used by the extractor. | | db\_extractor\_rows\_fetched | Counter | The number of rows fetched from a database, labeled on each query. | | db\_extractor\_rows\_uploaded | Counter | The total number of rows successfully uploaded to CDF. | | db\_extractor\_queries\_run | Counter | The number of successful queries. | | db\_extractor\_queries\_failed | Counter | The number of failed queries. | | db\_extractor\_connection\_num | Counter | The number of concurrent database connections active. This is the number of active queries. | | db\_extractor\_server\_num | Counter | The number of databases specified in the configuration. | | db\_extractor\_running | Counter | The state of the extractor as a numeric value. 1 indicates running and 0 indicates not running. | | db\_extractor\_info | | An information dictionary about the given extractor. This is the extractor's name and version. | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/db/db_server_requirements Hardware, software, and network requirements for running the Cognite DB extractor. You can install the extractor on a physical server, on a virtual machine on-premises, or in the cloud. ## Memory For the recommended default values in the configuration, the server needs a minimum of 1 GB RAM and an additional 256 GB HD for logging and storage. You can adjust the configuration to reduce memory usage. ## CPU For the recommended default values in the configuration, the server needs a minimum of 2-core CPUs. ## Software * Operating system: Windows 10, or later. * ODBC drivers: Windows-based ODBC drivers for any database you want to connect to using ODBC. ## Network access Network access with firewall openings to the database you're connecting to and the CDF project. ## CDF access You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. Learn more about authentication in the admin guide for OpenID Connect. # Set up the DB extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/db/db_setup Install and configure the Cognite DB extractor to connect to databases and load data into Cognite Data Fusion (CDF). ## Before you start * Assign access capabilities for the extractor to write data to the respective CDF destination resources. * If the database you're connecting to requires the extractor to use ODBC, download and install the ODBC drivers for your database. To create and monitor the extractor as an **integration** (public preview), see [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration). You can also navigate to Data fusion > Integrate > Extractors > Cognite DB extractor to download packages and see supported sources. * Review the server requirements for the extractor. * Create a configuration file according to the configuration settings. The file must be in YAML format. ## Connect to a database ### Native The extractor has native support for some databases and doesn't require any additional drivers. ### ODBC To connect via ODBC, you must install an ODBC driver for your database system on the machine where you're running the extractor. Here are links to ODBC drivers for some source systems: * MS SQLServer * MySQL * Oracle Consult the documentation for your database or contact the vendor if you need help finding ODBC drivers. ### ODBC connection strings ODBC uses connection strings to reference databases. The connection strings contain information about which ODBC driver to use, where to find the database, sign-in credentials, etc. ### Set up a Data Source Name (DSN) We recommend setting up a DSN for the database if you're running the extractor against an ODBC source on Windows. The Windows DSN system handles password storage instead of keeping it in the configuration file or as an environment variable. In addition, the connection strings will be less complex. Open the **ODBC Data Sources** tool on the machine you're running the extractor from. Select **Add** and the ODBC driver you want to use. In this example, we're configuring a PostgreSQL database. ODBC Data Source Administrator dialog showing available ODBC drivers including PostgreSQL Select **Finish**. Enter the connection information and the database name. The configuration dialog may differ depending on which database type you are configuring a DSN for. PostgreSQL ODBC Driver Setup dialog with connection configuration fields Select **Test** to verify that the information is correct, then **Save**. Use the simplified connection string in your configuration file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} databases: - name: my-postgres connection-string: 'DSN=MyPostgreSQL' ``` ## Run the extractor Download the `dbextractor-standalone-{VERSION}-win32.exe` file via the download links available from the Cognite DB extractor section on the **Extract data** page in CDF. Save the file in a folder. Open a command line window and run the file with a configuration file as an argument. In this example, the configuration file is named `config.yml` and saved in the same folder as the executable file: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} dbextractor-standalone--win32.exe ./config.yml ``` Download the `dbextractor-service-{VERSION}-win32.exe` file via the download links available from the Cognite DB extractor section on the **Extract data** page in CDF. Save the file to the same directory as a configuration file named `config.yml`. This must be the exact name of the file. As an administrator, open the command line window in the folder you placed the executable file and the configuration file and run the following command: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .\dbextractor-winservice--win32.exe install ``` Open the **Services** app in Windows and find the **Cognite DB Extractor Executor** service. Right-click the service and select **Properties**. Configure the service according to your requirements. Use a Docker container to run the DB extractor on Mac OS or Linux systems. This article describes how to create file-specific bindings with the container. You should consider sharing a folder if a deployment uses more than one file, for example, logs and local state files. ### ODBC only: Create a custom docker image If you're using ODBC to connect to your database, you first need to create a docker image that contains the ODBC driver for your database. Cognite ships a base image for the DB extractor that contains the extractor and ODBC libraries, but not specific drivers. This example uses a PostgreSQL database. This example uses PostgreSQL since the driver is easily available in Debian's package repository. When you're connecting to a PostgreSQL database, use the native PostgreSQL support in the DB extractor. Create a Dockerfile and extend the `cognite/db-extractor-base` image. We recommend locking the version to a major release. Then, install your ODBC drivers. For example, for PostgreSQL: ```dockerfile theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} FROM cognite/db-extractor-base:3 RUN apt-get update \ && apt-get install -y odbc-postgresql \ && apt-get clean -y \ && rm -rf /var/lib/apt/lists/* \ && rm -rf /tmp/* ``` Save the Docker file with a descriptive name. In this example, we use `postgres.Dockerfile`. Build the Docker image. Give the image a tag with the `-t` argument to make it easy to refer to the image. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} docker build -f postgres.Dockerfile -t db-extractor-postgres . ``` Replace `postgres.Dockerfile` with the name of your Docker file. Replace `db-extractor-postgres` with the tag you want to assign to this Docker image. Run this image to create a Docker container: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} docker run db-extractor-postgres ``` Replace `db-extractor-postgres` with the tag you created above. **Run on a local host on Docker** When a Docker container runs, the `localhost` address points to the container, not the host. This creates an issue if the container runs against a local database. For development and test environments, you can use the `host.docker.internal` DNS name to resolve to the host. Don't use this approach for production environments. You also need the local database to accept connections from outside the localhost for the container to access the database connection. ## Load data incrementally If the database table has a column containing an incremental field, you can set up the extractor to only process new or updated rows since the last extraction. An incremental field is a field that increases for new entries, such as time stamps for the latest update or insertions if the rows never change, or a numerical index. To load data incrementally for a query: Include the `state-store` parameter in the `extractor` section of the configuration. Include the `incremental-field` and `initial-start` parameters in the query configuration. Update the SQL query with a `WHERE` statement using `{incremental-field}` and `{start-at}`. For example: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT * FROM table WHERE {incremental-field} >= '{start-at}' ORDER BY {incremental-field} ASC ``` If the query returns many rows, consider paging the results with the `LIMIT` statement. Be aware that this demands a well-defined paging strategy. Consider this example: ```sql theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT * FROM data WHERE {incremental-field} >= '{start-at}' ORDER BY {incremental-field} ASC LIMIT 10 ``` **Multiple rows with identical values in the incremental field** If 10 or more rows have the same time, the `start-at` field won't be updated since the largest value of the incremental field didn't increase in the run. The next run will start with the same value, and the extractor is stuck in a loop. However, changing the query from `>=` to `>` can cause data loss if many rows have the same value for the incremental field. Therefore, set a sufficiently high limit if there are duplicates in the incremental field and you're using result paging. There might be a slight boost in the extractor runtime if the database has an index on the incremental field. However, for a standard run of the DB extractor, network delays are often the main bottleneck for performance, so usually, the difference is minimal. # Ingest data from Snowflake with OAuth Source: https://docs.cognite.com/cdf/integration/guides/extraction/db/db_snowflake_oauth_setup Configure the DB extractor to ingest data from Snowflake using OAuth 2.0 authentication with Microsoft Entra ID. ## Before you start Connect the DB extractor to a Snowflake database. Follow the steps in the Knowledge Base Article from Snowflake about creating external OAuth tokens using Azure AD for the OAuth client. When the connection is set up, use these attributes to configure the DB extractor: * Client ID * Client secret * Token endpoint * OAuth scopes ## Configuration example This is a configuration example for a Snowflake database using OAuth. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} databases: - name: Snowflake_OAuth type: snowflake authentication_type: oauth # Credentials client-id: ${Client ID} client-secret: ${Client Secret} access_token_generate_url: ${Token Endpoint} oauth_scopes: ${OAuth Scopes} # Account Info account: ${Snowflake Account ID} organization: ${Snowflake Organization ID} database: ${Snowflake Database} schema: ${Snowflake Schema} compute-warehouse: ${Snowflake Compute Warehouse} ``` # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/extraction/db/db_troubleshooting Common issues and solutions for the Cognite DB extractor. ## The extractor is receiving 401 errors from CDF HTTP error 401 means that the service did not recognize the authentication the client provided. If the Cognite API returns 401 errors to the extractor, the extractor successfully obtained an authentication token, but the token is not valid for the project in question. If this happens, the configured `project` or `host` in the `cognite` section is usually incorrect. ## The extractor is not setting the data set when using remote configuration If you set up the extractor to run with remote configuration, only the `host`, `project`, `idp-authentication`, and `extraction-pipeline` fields in the `cognite` section are read from the file on the local machine. Other fields in the `cognite` section, such as `data-set`, must be added to the configuration file in the extraction pipelines. # About the Cognite EDM extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/edm Extract data from Landmark Engineers Data Model (EDM) server to the Cognite Data Fusion (CDF) staging area through OData protocol from DecisionSpace Integration Server (DSIS). You can set up the extractor for both **full-load** extractions and **incremental** extractions. You load data incrementally by adding the [delta-supported EDM entities](/cdf/integration/guides/extraction/edm/edm_entities) to the `entities` configuration parameter and set a schedule for checking incremental EDM entities with the `gracePeriod` configuration parameter. The extractor creates a CDF RAW table for each EDM entity. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/edm/edm_configuration Configure the EDM extractor with YAML file settings for Cognite, logging, EDM connection, and entity extraction parameters. To configure the EDM extractor, you must create a configuration file. The file must be in [YAML format](https://yaml.org). You can use the [sample minimal configuration file](https://github.com/cognitedata/subsurface-extractor-config-samples/blob/main/edm-extractor/edm-config-sample.yml) as a starting point for your configuration settings. You can set parameters through environment variables using the pattern ` `. The environment variables override values set in the configuration file (including default values for parameters not listed in the configuration file). For example, setting the `COGNITE_PROJECT` environment variable overrides the `project` parameter under the `Cognite` section (the matching is case insensitive). ## Cognite Include the `cognite` section to configure which CDF project the extractor loads data into and how to connect to the project. This section is mandatory and should always contain the project and authentication configuration. | Parameter | Description | | - | - | | `clientId` | Insert the CDF client ID. This is mandatory if you're using OIDC authentication. | | `clientSecret` | Insert the CDF client secret for your CDF project. This is mandatory if you're using OIDC authentication. | | `tokenUrl` | Insert the token URL. This is mandatory if you're using OIDC authentication. You'll find further details on the OIDC token URL in [authorization and groups](/cdf/access/concepts/best_practices_oidc#authorization-and-groups). | | `apiUrl` | Insert the base URL of the CDF project. | | `project` | Insert the CDF project name you want to ingest data into. This field is required. | | `rawDatabaseName` | Enter the name of the CDF RAW database to extract data into. If it doesn't already exist, the extractor creates the database. | | `queueBufferSize` | Enter the size for the queue between extraction and writing objects to CDF RAW. The default value is 500. A larger queue will allow for buffering more extracted entities from EDM, but lead to higher memory usage. | ## Logging Include the `logging` section to set up logging to a standard output, such as a terminal window. | Parameter | Description | | - | - | | `level` | Select the verbosity level for console logging. Valid options are `trace`, `debug`, `info`, `warn`, and `error`. | | `cdfLevel` | Set the log level for the CDF Java SDK. Valid options are `trace`, `debug`, `info`, `warn`, and `error`. This is an optional value. The default value is `warn`. | ## EDM Include the `edm` section to configure the parameters needed to connect to your EDM instance. | Parameter | Description | | - | - | | `dsisAuthServer` | If you set `dsisAuthType` to `token`, enter the URL for DSIS authentication | | `dsisAuthType` | Select how to authenticate to DSIS. Valid options are `token` and `basic`.
  • `token` expects the bearer token from the DSIS authentication server.
  • `basic` uses `dsisUsername` and `dsisPassword` to authenticate.
  • | | `verifyDsisAuthServerCertificate` | Set to `false` to disable certificate verification for the DSIS auth Server specified with the `dsisAuthServer` option. This field is optional and verification defaults to `true`. | | `gracePeriod` | Enter the time in seconds between each scan for changes to [incremental entities](/cdf/integration/guides/extraction/edm/edm_entities#incremental-entities). The default value is 180. This field is optional. | | `crontabComplete` | Set up a [cron](https://en.wikipedia.org/wiki/Cron) schedule for starting the extraction of complete entities. | | `runOnStartup` | Set to `true` to run a complete extraction on startup bypassing any schedule set in `crontabComplete`. This is typically used during development. The default value is `false`. This field is optional. | | `dsisEdmPath` | Insert the path to the EDM server. | | `dsisPathCartoServices` | Insert the path to the EDM Carto services to extract Carto entities from EDM, such as CD\_CARTO. | | `dsisParallelism` | Insert the number of threads to extract data from EDM. Be aware that a high volume load on the EDM server may cause a crash. The default value is 2. This field is optional. | | `processingParallelism` | Insert the number of threads to process and upload extracted entities to CDF RAW. The default value is 4. This field is optional. | | `startPadding` | Extractor internal parameter. **Do not change**. | | `measurementSystem` | Enter the measurement system defined in EDM/DSIS. If you set a value, the `measurementsystem` will be appended to all queries. The default value is blank. This field is optional. | | `connectTimeout` | Enter the maximum time in seconds to wait for a connection to DSIS. The default value is 5. This field is optional. | | `readTimeout` | Set the maximum timeout in seconds for read requests to DSIS. This field is optional. If you don't enter a value or the value is less then 120, the default value is set to 120. | | `entitiesCompleteExtraction` | List the EDM entities to extract completely on each run, even if the entities are incremental. This can be used as a workaround if EDM is missing `create_date` and `update_date` on incremental entities. | | `oDataConsumerDebug` | Enable debug logging of EDM traffic for troubleshooting. Valid strings are `ALL`, `REQUEST_HEADER`, `REQUEST_FULL`, `RESPONSE_HEADER`, `RESPONSE_FULL`, `OFF`. This field is optional. | | `cartoSync` | Set to `true` to extract the EDM cartography database used (entities `carto` and `carto alias`). These will be extracted into CDF RAW tables named `Carto` and `CartoAlias`, respectively. | | `dsisBaseUrl` | Insert the base URL of the DSIS. The extractor combines `dsisBaseUrl`, `dsisEdmPath`, `dsisEdmVdb`, and `dsisEdmProject` to connect to your EDM instance. This is a required field. | | `dsisEdmVdb` | Enter the name of the EDM virtual database. This is a required field. | | `dsisEdmProject` | Enter the name of the EDM project. The project name `ALL_PROJECTS` can be used to query across all projects in EDM. This is a required field. | | `dsisAuthRealm` | Enter the name of the EDM authentication REALM. This is a required field. | | `dsisAuthClientId` | Enter the DSIS client ID. This is a required field. | | `dsisUsername` | Enter a valid DSIS username. This is a required field. | | `dsisPassword` | Enter a valid DSIS password. This is a required field. | | `entities` | List the EDM entities you want to extract. See also the `entities` section below. | | `consistencySchedule` | Set up a [cron](https://en.wikipedia.org/wiki/Cron) schedule for triggering a consistency check of the state of RAW compared to EDM to mark deleted objects in EDM as deleted in RAW. | | `consistencyOnStartup` | Set to `true` to run a consistency check between the state in RAW and EDM on startup bypassing any schedule set in `consistencySchedule`. | ## Entities Include the `entities` section to list the entities you want to extract from the EDM server to CDF. You'll find the list of supported EDM entities [here](/cdf/integration/guides/extraction/edm/edm_entities). | Parameter | Description | | - | - | | `name` | Enter the name of the EDM entity, for instance, *CD\_WELL*. This is a required field. | | `filter` | OData-based custom filters for EDM entity level. Use this to filter unwanted records from EDM. | | `includeFields` | List of properties on the entity to include in the extraction. If not specified, all fields are included. This field can't be combined with `excludefields`. | | `excludeFields` | List of properties to exclude from the extraction of an entity. Don't use this parameter together with `includeFields`. | | `overrideIncremental` | This parameter is only relevant when the entity is listed in the `entitiesCompleteExtraction` parameter and confirms for the extractor that it should both extract incrementally and complete. The entry under `entitiesCompleteExtraction` parameter should have a filter specified to only select entities that are missed by the incremental extraction. Use in combination with a filter to get all entities with create/update\_date set to NULL. See the sample configuration file. Use this parameter for special cases only. By default, this parameter is disabled. | | `batchSize` | The number of entities to retrieve per request. The default value is 8, or 50 for the `related` parameter. Some heavy entities, such as CD\_WELL, have further configuration in the extractor to reduce the number of entities to retrieve in a batch to reduce the load on the EDM server. This value will override these built-in standard values. This field is optional. | | `related` | Set to `true` to include related entities in the object extracted. By default set to `true`. | | `relatedList` | Set to an optional list of related entities to include. If not set, all related entities are included. | | `consistencyCheck` | Set to `false` to exclude an entity from the set of entities that are consistency checked periodically (according to the schedule set in `consistencySchedule`). The consistency check verifies whether all extracted objects in RAW still exist in EDM. If they don't exist, they're marked as deleted in RAW. The default value is `true`. | # EDM data entities Source: https://docs.cognite.com/cdf/integration/guides/extraction/edm/edm_entities Comprehensive reference of supported EDM entities for the Cognite EDM extractor, including incremental and non-incremental entity types with their extraction specifications. The Cognite EDM extractor translates EDM data entities to store and maintain in CDF. Data is retrieved using the EDM native data model, and the extracted objects will match the EDM OData entity schema. This translation is automatically done inside the Cognite EDM extractor. There are two EDM entities categories: * **Incremental entities** have a timestamp field used by the extractor to handle and control incremental runs. * **Non-incremental entities** don't have a timestamp field. These entities are always updated and inserted in full load into CDF. When extracting an entity, the extractor also retrieves any other entities it relates to that aren't in the incremental category (in a transactional manner). This means that the extracted entity written to RAW contains the full context of the entity, avoiding the need to join multiple RAW tables. The **Number of related entities** column in the list of entities specifies how many related entities an entity references. This can be overridden on an entity-by-entity basis (such as to reduce the load on EDM) by either turning off getting related entities completely or specifying which related entities to include in the configuration through the `includeFields` or `excludeFields`. The Cognite EDM extractor supports these EDM entities: ## Incremental entities | **Entity name** | **Protobuf name** | **Extraction type** | **Number of related entities** | | :- | :- | :- | -: | | CD\_ACC\_CATALOG | cd.CdAccCatalog | Incremental | 0 | | CD\_AIR\_PACKAGE | cd.CdAirPackage | Incremental | 0 | | CD\_ASSEMBLY | cd.CdAssembly | Incremental | 10 | | CD\_ASSEMBLY\_COMP | cd.CdAssemblyComp | Incremental | 22 | | CD\_ASSEMBLY\_COMP\_INNER | cd.CdAssemblyCompInner | Incremental | 0 | | CD\_ASSEMBLY\_INNER | cd.CdAssemblyInner | Incremental | 0 | | CD\_ATTACHMENT | cd.CdAttachment | Incremental | 0 | | CD\_ATTACHMENT\_JOURNAL | cd.CdAttachmentJournal | Incremental | 0 | | CD\_BED\_IMPELLER\_CATALOG | cd.CdBedImpellerCatalog | Incremental | 0 | | CD\_BHA\_COMP\_CBL\_STAB | cd.CdBhaCompCblStab | Incremental | 0 | | CD\_BHA\_COMP\_MWD | cd.CdBhaCompMwd | Incremental | 1 | | CD\_BHA\_COMP\_NOZZLE | cd.CdBhaCompNozzle | Incremental | 0 | | CD\_BHA\_COMP\_NOZZLE\_INNER | cd.CdBhaCompNozzleInner | Incremental | 0 | | CD\_BHA\_COMP\_STAB | cd.CdBhaCompStab | Incremental | 0 | | CD\_BHA\_COMP\_STAB\_INNER | cd.CdBhaCompStabInner | Incremental | 0 | | CD\_BIT\_CATALOG | cd.CdBitCatalog | Incremental | 0 | | CD\_BIT\_NOZZLE\_CATALOG | cd.CdBitNozzleCatalog | Incremental | 0 | | CD\_CASE | cd.CdCase | Incremental | 83 | | CD\_CASING\_SCRAPER\_CATALOG | cd.CdCasingScraperCatalog | Incremental | 0 | | CD\_CASING\_SHOE\_CATALOG | cd.CdCasingShoeCatalog | Incremental | 0 | | CD\_CATALOG | cd.CdCatalog | Incremental | 1 | | CD\_CBL\_CATALOG | cd.CdCblCatalog | Incremental | 0 | | CD\_CBL\_STAB\_CATALOG | cd.CdCblStabCatalog | Incremental | 0 | | CD\_CENTRALIZER\_CATALOG | cd.CdCentralizerCatalog | Incremental | 1 | | CD\_CLASS | cd.CdClass | Incremental | 0 | | CD\_COMPLETION | cd.CdCompletion | Incremental | 10 | | CD\_COMPLETIONS\_CATALOG | cd.CdCompletionsCatalog | Incremental | 0 | | CD\_CONTRACTOR | cd.CdContractor | Incremental | 0 | | CD\_CSG\_CATALOG | cd.CdCsgCatalog | Incremental | 0 | | CD\_CT\_CATALOG | cd.CdCtCatalog | Incremental | 0 | | CD\_DATUM | cd.CdDatum | Incremental | 0 | | CD\_DC\_CATALOG | cd.CdDcCatalog | Incremental | 0 | | CD\_DEFINITIVE\_SURVEY\_HEADER | cd.CdDefinitiveSurveyHeader | Incremental | 5 | | CD\_DEPTH\_INTERVAL\_HISTORY | cd.CdDepthIntervalHistory | Incremental | 0 | | CD\_DP\_CATALOG | cd.CdDpCatalog | Incremental | 0 | | CD\_ECC\_CATALOG | cd.CdEccCatalog | Incremental | 0 | | CD\_EXPLORER\_FOLDER | cd.CdExplorerFolder | Incremental | 1 | | CD\_FACILITY | cd.CdFacility | Incremental | 0 | | CD\_FISHING\_CATALOG | cd.CdFishingCatalog | Incremental | 0 | | CD\_FLUID | cd.CdFluid | Incremental | 4 | | CD\_FORMATION\_PICK | cd.CdFormationPick | Incremental | 0 | | CD\_FRAC\_GRADIENT\_GROUP | cd.CdFracGradientGroup | Incremental | 1 | | CD\_GRADE | cd.CdGrade | Incremental | 1 | | CD\_HOLE\_SECT\_GROUP | cd.CdHoleSectGroup | Incremental | 2 | | CD\_HW\_CATALOG | cd.CdHwCatalog | Incremental | 0 | | CD\_INTERPRETATION | cd.CdInterpretation | Incremental | 0 | | CD\_JAR\_CATALOG | cd.CdJarCatalog | Incremental | 0 | | CD\_LESSON | cd.CdLesson | Incremental | 5 | | CD\_LOCATION | cd.CdLocation | Incremental | 18 | | CD\_MATERIAL | cd.CdMaterial | Incremental | 0 | | CD\_MOTOR\_CATALOG | cd.CdMotorCatalog | Incremental | 0 | | CD\_MUD\_PUMP\_CATALOG | cd.CdMudPumpCatalog | Incremental | 0 | | CD\_MWD\_CATALOG | cd.CdMwdCatalog | Incremental | 0 | | CD\_NEAR\_BIT\_REAMER\_CATALOG | cd.CdNearBitReamerCatalog | Incremental | 0 | | CD\_NOZZLE\_CATALOG | cd.CdNozzleCatalog | Incremental | 0 | | CD\_PAD\_OP | cd.CdPadOp | Incremental | 3 | | CD\_PERFGUN\_CATALOG | cd.CdPerfgunCatalog | Incremental | 0 | | CD\_POLICY | cd.CdPolicy | Incremental | 5 | | CD\_PORE\_PRESSURE\_GROUP | cd.CdPorePressureGroup | Incremental | 1 | | CD\_PRESSURE\_SURVEY | cd.CdPressureSurvey | Incremental | 6 | | CD\_PROJECT | cd.CdProject | Incremental | 1 | | CD\_PROP\_CONNECT\_CATALOG | cd.CdPropConnectCatalog | Incremental | 0 | | CD\_REALTIME\_INSTRUMENT | cd.CdRealtimeInstrument | Incremental | 0 | | CD\_REAMER\_SUB\_CATALOG | cd.CdReamerSubCatalog | Incremental | 0 | | CD\_REGION | cd.CdRegion | Incremental | 0 | | CD\_RIG | cd.CdRig | Incremental | 3 | | CD\_RIG\_ANCHOR | cd.CdRigAnchor | Incremental | 0 | | CD\_RIG\_BOILER | cd.CdRigBoiler | Incremental | 0 | | CD\_RIG\_BOP | cd.CdRigBop | Incremental | 1 | | CD\_RIG\_CENTRIFUGE | cd.CdRigCentrifuge | Incremental | 0 | | CD\_RIG\_DEGASSER | cd.CdRigDegasser | Incremental | 0 | | CD\_RIG\_HYDROCLONE | cd.CdRigHydroclone | Incremental | 0 | | CD\_RIG\_MOTOR | cd.CdRigMotor | Incremental | 0 | | CD\_RIG\_PIT | cd.CdRigPit | Incremental | 0 | | CD\_RIG\_PUMP | cd.CdRigPump | Incremental | 0 | | CD\_RIG\_SHAKER | cd.CdRigShaker | Incremental | 1 | | CD\_RIG\_SURFACE | cd.CdRigSurface | Incremental | 0 | | CD\_RSS\_CATALOG | cd.CdRssCatalog | Incremental | 0 | | CD\_SCENARIO | cd.CdScenario | Incremental | 30 | | CD\_SCREEN\_CATALOG | cd.CdScreenCatalog | Incremental | 0 | | CD\_SITE | cd.CdSite | Incremental | 3 | | CD\_SITE\_LEASE | cd.CdSiteLease | Incremental | 0 | | CD\_SITE\_STATUS | cd.CdSiteStatus | Incremental | 0 | | CD\_STAB\_CATALOG | cd.CdStabCatalog | Incremental | 0 | | CD\_STRAT\_UNIT | cd.CdStratUnit | Incremental | 0 | | CD\_SURFACE\_HEADER | cd.CdSurfaceHeader | Incremental | 1 | | CD\_SURVEY\_HEADER | cd.CdSurveyHeader | Incremental | 2 | | CD\_SURVEY\_PROGRAM | cd.CdSurveyProgram | Incremental | 0 | | CD\_SURVEY\_STATION | cd.CdSurveyStation | Incremental | 2 | | CD\_SURVEY\_TOOL | cd.CdSurveyTool | Incremental | 1 | | CD\_TEMP\_GRADIENT\_GROUP | cd.CdTempGradientGroup | Incremental | 1 | | CD\_TEST | cd.CdTest | Incremental | 11 | | CD\_WELL | cd.CdWell | Incremental | 8 | | CD\_WELLBORE | cd.CdWellbore | Incremental | 28 | | CD\_WELLBORE\_ALIAS | cd.CdWellboreAlias | Incremental | 0 | | CD\_WELLBORE\_FORMATION | cd.CdWellboreFormation | Incremental | 1 | | CD\_WELLBORE\_INTEREST | cd.CdWellboreInterest | Incremental | 0 | | CD\_WELLBORE\_REMARKS | cd.CdWellboreRemarks | Incremental | 0 | | CD\_WELLBORE\_STATUS | cd.CdWellboreStatus | Incremental | 0 | | CD\_WELLBORE\_ZONE | cd.CdWellboreZone | Incremental | 3 | | CD\_WELL\_ALIAS | cd.CdWellAlias | Incremental | 0 | | CD\_WELL\_ALL | cd.CdWellAll | Incremental | 0 | | CD\_WELL\_APPROVAL | cd.CdWellApproval | Incremental | 0 | | CD\_WELL\_COST\_REDUCTION | cd.CdWellCostReduction | Incremental | 0 | | CD\_WELL\_LEASE | cd.CdWellLease | Incremental | 0 | | CD\_WELL\_MAASP | cd.CdWellMaasp | Incremental | 0 | | CD\_WELL\_REMARKS | cd.CdWellRemarks | Incremental | 0 | | CD\_WELL\_STATUS | cd.CdWellStatus | Incremental | 0 | | CD\_WELL\_STATUS\_HISTORY | cd.CdWellStatusHistory | Incremental | 0 | | CD\_ZONATION\_SCHEME | cd.CdZonationScheme | Incremental | 0 | | CD\_ZONE | cd.CdZone | Incremental | 0 | | CD\_ZONE\_STRUCTURE | cd.CdZoneStructure | Incremental | 0 | | DM\_AFE | dm.DmAfe | Incremental | 5 | | DM\_AFE\_COST | dm.DmAfeCost | Incremental | 0 | | DM\_ANCHOR\_TEST | dm.DmAnchorTest | Incremental | 0 | | DM\_BHA\_RUN | dm.DmBhaRun | Incremental | 1 | | DM\_CASING | dm.DmCasing | Incremental | 1 | | DM\_CONTRACT | dm.DmContract | Incremental | 3 | | DM\_CR\_CONSTRUCTION | dm.DmCrConstruction | Incremental | 12 | | DM\_CR\_LEASE | dm.DmCrLease | Incremental | 0 | | DM\_CR\_SCOUT | dm.DmCrScout | Incremental | 10 | | DM\_DAILYCOST | dm.DmDailycost | Incremental | 1 | | DM\_EVENT | dm.DmEvent | Incremental | 29 | | DM\_EVENT\_APPROVAL | dm.DmEventApproval | Incremental | 0 | | DM\_EVENT\_REMARKS | dm.DmEventRemarks | Incremental | 0 | | DM\_EVENT\_STATUS | dm.DmEventStatus | Incremental | 0 | | DM\_EVENT\_TASKS | dm.DmEventTasks | Incremental | 0 | | DM\_HOTNOTE | dm.DmHotnote | Incremental | 0 | | DM\_INCIDENT | dm.DmIncident | Incremental | 2 | | DM\_OPER\_EQUIP\_FAIL | dm.DmOperEquipFail | Incremental | 1 | | DM\_OTHER\_CONTRACT | dm.DmOtherContract | Incremental | 0 | | DM\_REGULATORY\_AUDIT | dm.DmRegulatoryAudit | Incremental | 0 | | DM\_REPORT\_JOURNAL | dm.DmReportJournal | Incremental | 2 | | DM\_RIG\_MOVE | dm.DmRigMove | Incremental | 3 | | DM\_RIG\_OPERATION | dm.DmRigOperation | Incremental | 1 | | DM\_RISER | dm.DmRiser | Incremental | 1 | | DM\_RISERLINE\_OP | dm.DmRiserlineOp | Incremental | 1 | | DM\_VENT\_FLOW\_TEST | dm.DmVentFlowTest | Incremental | 0 | | DM\_WASTE\_TRANSFER | dm.DmWasteTransfer | Incremental | 1 | | DM\_WELL\_PLAN\_OFFSET | dm.DmWellPlanOffset | Incremental | 2 | | DP\_PROJECT\_TARGET | dp.DpProjectTarget | Incremental | 5 | | DP\_TEMPLATE | dp.DpTemplate | Incremental | 0 | | IWP\_REPORT | iw\.IwpReport | Incremental | 0 | | IWP\_TEMPLATE | iw\.IwpTemplate | Incremental | 0 | | MD\_DESKTOP\_FORMATS | md.MdDesktopFormats | Incremental | 0 | | MD\_SITE\_ATTR\_DICTIONARY | md.MdSiteAttrDictionary | Incremental | 0 | | MD\_SITE\_CONFIG\_FILES | md.MdSiteConfigFiles | Incremental | 0 | | MD\_SITE\_DEX\_TEMPLATE | md.MdSiteDexTemplate | Incremental | 1 | | MD\_SITE\_DV\_RULE\_BOOKS | md.MdSiteDvRuleBooks | Incremental | 1 | | MD\_SYS\_DV\_RULE\_BOOKS | md.MdSysDvRuleBooks | Incremental | 1 | | MD\_SYS\_PICKLIST | md.MdSysPicklist | Incremental | 2 | | MD\_SYS\_PICKLIST\_SCRIPT | md.MdSysPicklistScript | Incremental | 1 | | MD\_SYS\_PICKLIST\_TABLE\_NAMES | md.MdSysPicklistTableNames | Incremental | 0 | | MD\_SYS\_SCRIPT | md.MdSysScript | Incremental | 1 | | MD\_SYS\_VERSION | md.MdSysVersion | Incremental | 0 | | MD\_SYS\_VERSION\_HISTORY | md.MdSysVersionHistory | Incremental | 0 | | NT\_WELLCASE | nt.NtWellcase | Incremental | 1 | | RT\_LOG\_CURVE\_HEADER | rt.RtLogCurveHeader | Incremental | 2 | | RT\_LOG\_HEADER | rt.RtLogHeader | Incremental | 0 | | TU\_DLS\_OVERRIDE | tu.TuDlsOverride | Incremental | 0 | | TU\_DLS\_OVERRIDE\_GROUP | tu.TuDlsOverrideGroup | Incremental | 0 | | TU\_ZONE\_PRESSURE | tu.TuZonePressure | Incremental | 0 | | TU\_ZONE\_PRESSURE\_GROUP | tu.TuZonePressureGroup | Incremental | 0 | | WP\_CASE\_RESULT | wp.WpCaseResult | Incremental | 0 | ## Non-incremental entities These entities don't have a `create/update` timestamp field, and an extractor can't detect changes without downloading all the content. | **Entity name** | **Protobuf name** | **Extraction types** | **Number of Related Entities** | | :- | :- | :- | -: | | CD\_AIR\_PACKAGE\_RIG\_LINK | cd.CdAirPackageRigLink | Complete | 0 | | CD\_ASSEMBLY\_COMP\_FAILURE | cd.CdAssemblyCompFailure | Complete | 0 | | CD\_ASSEMBLY\_COMP\_FISH\_TOOL | cd.CdAssemblyCompFishTool | Complete | 0 | | CD\_ASSEMBLY\_COMP\_STATUS | cd.CdAssemblyCompStatus | Complete | 0 | | CD\_ASSEMBLY\_DETAIL | cd.CdAssemblyDetail | Complete | 0 | | CD\_ASSEMBLY\_ID\_MIN | cd.CdAssemblyIdMin | Complete | 0 | | CD\_ASSEMBLY\_NOTE | cd.CdAssemblyNote | Complete | 0 | | CD\_ASSEMBLY\_STATUS | cd.CdAssemblyStatus | Complete | 0 | | CD\_ASSEMBLY\_SUB | cd.CdAssemblySub | Complete | 0 | | CD\_BHA\_COMP\_BIT | cd.CdBhaCompBit | Complete | 0 | | CD\_BHA\_COMP\_BIT\_INNER | cd.CdBhaCompBitInner | Complete | 0 | | CD\_BHA\_COMP\_DP\_HW | cd.CdBhaCompDpHw | Complete | 0 | | CD\_BHA\_COMP\_DP\_HW\_INNER | cd.CdBhaCompDpHwInner | Complete | 0 | | CD\_BHA\_COMP\_ECCSTAB | cd.CdBhaCompEccstab | Complete | 0 | | CD\_BHA\_COMP\_JAR | cd.CdBhaCompJar | Complete | 0 | | CD\_BHA\_COMP\_JAR\_INNER | cd.CdBhaCompJarInner | Complete | 0 | | CD\_BHA\_COMP\_MOTOR | cd.CdBhaCompMotor | Complete | 1 | | CD\_BHA\_COMP\_MOTOR\_INNER | cd.CdBhaCompMotorInner | Complete | 0 | | CD\_BHA\_MOTOR\_COMP | cd.CdBhaMotorComp | Complete | 1 | | CD\_BHA\_MWD\_COMP | cd.CdBhaMwdComp | Complete | 0 | | CD\_BUSINESS\_ASSOCIATE | cd.CdBusinessAssociate | Complete | 1 | | CD\_BUSINESS\_ASSOCIATION | cd.CdBusinessAssociation | Complete | 1 | | CD\_CASE\_ASSEMBLY | cd.CdCaseAssembly | Complete | 0 | | CD\_CASE\_TEMP\_GRADIENT | cd.CdCaseTempGradient | Complete | 0 | | CD\_CEMENTPROP | cd.CdCementprop | Complete | 0 | | CD\_CEMENT\_FLUID | cd.CdCementFluid | Complete | 4 | | CD\_CEMENT\_FLUID\_ADD | cd.CdCementFluidAdd | Complete | 1 | | CD\_CEMENT\_FLUID\_SCHEDULE | cd.CdCementFluidSchedule | Complete | 1 | | CD\_CEMENT\_JOB | cd.CdCementJob | Complete | 5 | | CD\_CEMENT\_PLUG\_STATUS | cd.CdCementPlugStatus | Complete | 1 | | CD\_CEMENT\_SLURRY\_TEST | cd.CdCementSlurryTest | Complete | 1 | | CD\_CEMENT\_STAGE | cd.CdCementStage | Complete | 3 | | CD\_CENTRALIZER\_CAT\_FORCE | cd.CdCentralizerCatForce | Complete | 0 | | CD\_CHANGE\_HISTORY\_JOURNAL | cd.CdChangeHistoryJournal | Complete | 0 | | CD\_COMPLETION\_FLUID | cd.CdCompletionFluid | Complete | 0 | | CD\_COMPLETION\_INTERVENT | cd.CdCompletionIntervent | Complete | 0 | | CD\_COMPLETION\_NOTES | cd.CdCompletionNotes | Complete | 0 | | CD\_COMPLETION\_RETRIEVABLE | cd.CdCompletionRetrievable | Complete | 0 | | CD\_COMPLETION\_STATUS\_HISTORY | cd.CdCompletionStatusHistory | Complete | 0 | | CD\_COMPLETION\_TREATMENT | cd.CdCompletionTreatment | Complete | 0 | | CD\_COMPLETION\_UWI | cd.CdCompletionUwi | Complete | 0 | | CD\_COMPOSITE\_TOOL | cd.CdCompositeTool | Complete | 0 | | CD\_COMP\_SPEC\_FEATURE | cd.CdCompSpecFeature | Complete | 0 | | CD\_CONCUR | cd.CdConcur | Complete | 0 | | CD\_DEFINITIVE\_SURVEY\_STATION | cd.CdDefinitiveSurveyStation | Complete | 0 | | CD\_EXPLORER\_FOLDER\_MEMBER | cd.CdExplorerFolderMember | Complete | 0 | | CD\_EXTERNAL\_COMP\_ACC | cd.CdExternalCompAcc | Complete | 0 | | CD\_EXTERNAL\_COMP\_UMB | cd.CdExternalCompUmb | Complete | 0 | | CD\_EXTERNAL\_CONNECTION | cd.CdExternalConnection | Complete | 0 | | CD\_FORMATION\_INFLUX | cd.CdFormationInflux | Complete | 1 | | CD\_FORMATION\_INFLUX\_GROUP | cd.CdFormationInfluxGroup | Complete | 1 | | CD\_FRAC\_GRADIENT | cd.CdFracGradient | Complete | 0 | | CD\_GEO\_DATUM | cd.CdGeoDatum | Complete | 0 | | CD\_GEO\_ELLIPSOID | cd.CdGeoEllipsoid | Complete | 0 | | CD\_GEO\_SYSTEM | cd.CdGeoSystem | Complete | 1 | | CD\_GEO\_ZONE | cd.CdGeoZone | Complete | 1 | | CD\_GRADE\_SECT\_TYPE | cd.CdGradeSectType | Complete | 0 | | CD\_HOLE\_SECT | cd.CdHoleSect | Complete | 1 | | CD\_HOLE\_SECT\_STATUS | cd.CdHoleSectStatus | Complete | 1 | | CD\_LAND\_LEASE | cd.CdLandLease | Complete | 0 | | CD\_LESSON\_ACTION | cd.CdLessonAction | Complete | 1 | | CD\_LESSON\_ACTION\_STATUS | cd.CdLessonActionStatus | Complete | 1 | | CD\_LESSON\_BIBLIOGRAPHY | cd.CdLessonBibliography | Complete | 0 | | CD\_LESSON\_DISCUSSION | cd.CdLessonDiscussion | Complete | 0 | | CD\_LESSON\_KEYWORD | cd.CdLessonKeyword | Complete | 0 | | CD\_LESSON\_STATUS\_HISTORY | cd.CdLessonStatusHistory | Complete | 0 | | CD\_LIBRARY\_WELL\_LIST | cd.CdLibraryWellList | Complete | 0 | | CD\_LITHOLOGY\_CLASS | cd.CdLithologyClass | Complete | 0 | | CD\_LOCATION\_BC\_GRID | cd.CdLocationBcGrid | Complete | 0 | | CD\_LOCATION\_CARTER | cd.CdLocationCarter | Complete | 0 | | CD\_LOCATION\_CONGRESS | cd.CdLocationCongress | Complete | 0 | | CD\_LOCATION\_DLS | cd.CdLocationDls | Complete | 0 | | CD\_LOCATION\_FEDERAL | cd.CdLocationFederal | Complete | 0 | | CD\_LOCATION\_FPS | cd.CdLocationFps | Complete | 0 | | CD\_LOCATION\_GEODETIC | cd.CdLocationGeodetic | Complete | 0 | | CD\_LOCATION\_INTERNATIONAL | cd.CdLocationInternational | Complete | 0 | | CD\_LOCATION\_MAPPING | cd.CdLocationMapping | Complete | 0 | | CD\_LOCATION\_NARRATIVE | cd.CdLocationNarrative | Complete | 0 | | CD\_LOCATION\_NE | cd.CdLocationNe | Complete | 0 | | CD\_LOCATION\_NORTHSEA | cd.CdLocationNorthsea | Complete | 0 | | CD\_LOCATION\_NTS | cd.CdLocationNts | Complete | 0 | | CD\_LOCATION\_OCS | cd.CdLocationOcs | Complete | 0 | | CD\_LOCATION\_OHIO | cd.CdLocationOhio | Complete | 0 | | CD\_LOCATION\_ONTARIO\_OFFSHORE | cd.CdLocationOntarioOffshore | Complete | 0 | | CD\_LOCATION\_ONTARIO\_ONSHORE | cd.CdLocationOntarioOnshore | Complete | 0 | | CD\_LOCATION\_REFERENCE | cd.CdLocationReference | Complete | 0 | | CD\_LOCATION\_TEXAS | cd.CdLocationTexas | Complete | 0 | | CD\_NBK\_ANNULAR\_FLOW | cd.CdNbkAnnularFlow | Complete | 0 | | CD\_NBK\_DWU | cd.CdNbkDwu | Complete | 0 | | CD\_NBK\_FLUID\_COMPRESS | cd.CdNbkFluidCompress | Complete | 0 | | CD\_NBK\_GENERAL | cd.CdNbkGeneral | Complete | 0 | | CD\_NBK\_LEAK\_OFF\_TEST | cd.CdNbkLeakOffTest | Complete | 0 | | CD\_NBK\_LINEAR\_WEIGHT | cd.CdNbkLinearWeight | Complete | 0 | | CD\_NBK\_MIX\_FLUID | cd.CdNbkMixFluid | Complete | 0 | | CD\_NBK\_NOZZLES | cd.CdNbkNozzles | Complete | 0 | | CD\_NBK\_PIPE\_FLOW | cd.CdNbkPipeFlow | Complete | 0 | | CD\_NBK\_PUMP\_OUTPUT | cd.CdNbkPumpOutput | Complete | 0 | | CD\_NBK\_STRING\_ANALYSIS | cd.CdNbkStringAnalysis | Complete | 0 | | CD\_OPENING\_STATUS | cd.CdOpeningStatus | Complete | 1 | | CD\_OPENWIRE\_POLICY\_SETTING | cd.CdOpenwirePolicySetting | Complete | 0 | | CD\_OPENWIRE\_SETTING | cd.CdOpenwireSetting | Complete | 0 | | CD\_PAD | cd.CdPad | Complete | 2 | | CD\_PAD\_BULK | cd.CdPadBulk | Complete | 1 | | CD\_PAD\_MATERIAL\_TRANSFER | cd.CdPadMaterialTransfer | Complete | 2 | | CD\_PAD\_OP\_BULK\_TRAN | cd.CdPadOpBulkTran | Complete | 0 | | CD\_PAD\_OP\_WELL | cd.CdPadOpWell | Complete | 1 | | CD\_PAD\_OP\_WELL\_DAILYCOST | cd.CdPadOpWellDailycost | Complete | 1 | | CD\_PAD\_TRANSFER\_DETAIL | cd.CdPadTransferDetail | Complete | 1 | | CD\_PAD\_TRANSFER\_SUMMARY | cd.CdPadTransferSummary | Complete | 2 | | CD\_PERFORATE | cd.CdPerforate | Complete | 2 | | CD\_PERF\_ASSEMBLY\_LINK | cd.CdPerfAssemblyLink | Complete | 1 | | CD\_PERF\_CLUSTERS | cd.CdPerfClusters | Complete | 1 | | CD\_PERF\_INTERVAL | cd.CdPerfInterval | Complete | 2 | | CD\_POLICY\_REAL\_TIME\_CONFIG | cd.CdPolicyRealTimeConfig | Complete | 0 | | CD\_POLYLINE\_HEADER | cd.CdPolylineHeader | Complete | 1 | | CD\_POLYLINE\_POINT | cd.CdPolylinePoint | Complete | 1 | | CD\_PORE\_PRESSURE | cd.CdPorePressure | Complete | 0 | | CD\_PRESSURE\_AOF | cd.CdPressureAof | Complete | 1 | | CD\_PRESSURE\_AOF\_FLOW | cd.CdPressureAofFlow | Complete | 1 | | CD\_PRESSURE\_AWS | cd.CdPressureAws | Complete | 1 | | CD\_PRESSURE\_AWS\_AC | cd.CdPressureAwsAc | Complete | 1 | | CD\_PRESSURE\_SG | cd.CdPressureSg | Complete | 1 | | CD\_PRESSURE\_SG\_GRADIENT | cd.CdPressureSgGradient | Complete | 1 | | CD\_PRESSURE\_TRG | cd.CdPressureTrg | Complete | 0 | | CD\_PROJECT\_TARGET\_SITE\_LINK | cd.CdProjectTargetSiteLink | Complete | 0 | | CD\_PROJECT\_TARGET\_WB\_LINK | cd.CdProjectTargetWbLink | Complete | 0 | | CD\_PROJ\_TARG\_SCENARIO\_LINK | cd.CdProjTargScenarioLink | Complete | 0 | | CD\_PROJ\_TARG\_WELL\_LINK | cd.CdProjTargWellLink | Complete | 0 | | CD\_REAL\_TIME\_CONFIG | cd.CdRealTimeConfig | Complete | 0 | | CD\_RESERVOIR | cd.CdReservoir | Complete | 1 | | CD\_RESERVOIR\_STATUS | cd.CdReservoirStatus | Complete | 1 | | CD\_RIG\_BOP\_COMPONENT | cd.CdRigBopComponent | Complete | 0 | | CD\_ROD\_GUIDE | cd.CdRodGuide | Complete | 1 | | CD\_SCENARIO\_FORMATION\_LINK | cd.CdScenarioFormationLink | Complete | 0 | | CD\_SCENARIO\_SURFACE\_LINK | cd.CdScenarioSurfaceLink | Complete | 0 | | CD\_SITE\_LOCATION | cd.CdSiteLocation | Complete | 0 | | CD\_SITE\_OWNER | cd.CdSiteOwner | Complete | 1 | | CD\_SITE\_OWNER\_CONTACT | cd.CdSiteOwnerContact | Complete | 1 | | CD\_SITE\_TG | cd.CdSiteTg | Complete | 0 | | CD\_SLIP\_CRUSHING\_CATALOG | cd.CdSlipCrushingCatalog | Complete | 0 | | CD\_SURFACE\_TRI\_MESH | cd.CdSurfaceTriMesh | Complete | 0 | | CD\_SURVEY\_ANALYSIS\_LINK | cd.CdSurveyAnalysisLink | Complete | 1 | | CD\_SURVEY\_ANALYSIS\_LIST | cd.CdSurveyAnalysisList | Complete | 0 | | CD\_SURVEY\_GAUGE\_LINK | cd.CdSurveyGaugeLink | Complete | 1 | | CD\_TEMP\_GRADIENT | cd.CdTempGradient | Complete | 0 | | CD\_TEST\_ANALYSIS | cd.CdTestAnalysis | Complete | 4 | | CD\_TEST\_ANALYSIS\_LINK | cd.CdTestAnalysisLink | Complete | 1 | | CD\_TEST\_BEAM | cd.CdTestBeam | Complete | 0 | | CD\_TEST\_CAV | cd.CdTestCav | Complete | 0 | | CD\_TEST\_FLOW | cd.CdTestFlow | Complete | 0 | | CD\_TEST\_FLUID | cd.CdTestFluid | Complete | 0 | | CD\_TEST\_GAS | cd.CdTestGas | Complete | 0 | | CD\_TEST\_GAUGE | cd.CdTestGauge | Complete | 3 | | CD\_TEST\_GAUGE\_DATA | cd.CdTestGaugeData | Complete | 1 | | CD\_TEST\_GAUGE\_LINK | cd.CdTestGaugeLink | Complete | 1 | | CD\_TEST\_INTERP | cd.CdTestInterp | Complete | 1 | | CD\_TEST\_METER | cd.CdTestMeter | Complete | 2 | | CD\_TEST\_METER\_FLUID | cd.CdTestMeterFluid | Complete | 1 | | CD\_TEST\_METER\_GAS | cd.CdTestMeterGas | Complete | 1 | | CD\_TEST\_OIL | cd.CdTestOil | Complete | 2 | | CD\_TEST\_OIL\_DIST | cd.CdTestOilDist | Complete | 1 | | CD\_TEST\_OIL\_VIS | cd.CdTestOilVis | Complete | 1 | | CD\_TEST\_SWAB | cd.CdTestSwab | Complete | 0 | | CD\_TEST\_WATER | cd.CdTestWater | Complete | 0 | | CD\_VERTICAL\_SECTION | cd.CdVerticalSection | Complete | 0 | | CD\_VG\_METER | cd.CdVgMeter | Complete | 0 | | CD\_WBO\_COMP\_LINK | cd.CdWboCompLink | Complete | 0 | | CD\_WB\_REAL\_TIME\_CONFIG | cd.CdWbRealTimeConfig | Complete | 0 | | CD\_WELLBORE\_EXTERNAL | cd.CdWellboreExternal | Complete | 0 | | CD\_WELLBORE\_MARKER | cd.CdWellboreMarker | Complete | 0 | | CD\_WELLBORE\_OPENING | cd.CdWellboreOpening | Complete | 1 | | CD\_WELLBORE\_TYPE | cd.CdWellboreType | Complete | 0 | | CD\_WELLBORE\_UWI | cd.CdWellboreUwi | Complete | 0 | | CD\_WELLBORE\_ZONE\_PROP | cd.CdWellboreZoneProp | Complete | 0 | | CD\_WELLBORE\_ZONE\_STATUS | cd.CdWellboreZoneStatus | Complete | 0 | | CD\_WELLHEAD | cd.CdWellhead | Complete | 3 | | CD\_WELLHEAD\_COMP | cd.CdWellheadComp | Complete | 3 | | CD\_WELLHEAD\_COMPLETION\_LINK | cd.CdWellheadCompletionLink | Complete | 1 | | CD\_WELLHEAD\_COMP\_OUTLET | cd.CdWellheadCompOutlet | Complete | 1 | | CD\_WELLHEAD\_HANGER | cd.CdWellheadHanger | Complete | 2 | | CD\_WELLHEAD\_HANGER\_CONTROL | cd.CdWellheadHangerControl | Complete | 1 | | CD\_WELLHEAD\_TEST | cd.CdWellheadTest | Complete | 1 | | CD\_WELL\_EXTERNAL | cd.CdWellExternal | Complete | 0 | | CD\_WELL\_TG | cd.CdWellTg | Complete | 0 | | CD\_WELL\_UWI | cd.CdWellUwi | Complete | 0 | | CD\_WEQP\_BLOCK | cd.CdWeqpBlock | Complete | 0 | | CD\_WEQP\_CONV\_PUMP | cd.CdWeqpConvPump | Complete | 1 | | CD\_WEQP\_CPUMP | cd.CdWeqpCpump | Complete | 0 | | CD\_WEQP\_ESP | cd.CdWeqpEsp | Complete | 0 | | CD\_WEQP\_GAS\_LIFT\_MANDREL | cd.CdWeqpGasLiftMandrel | Complete | 3 | | CD\_WEQP\_GAS\_LIFT\_OPERATION | cd.CdWeqpGasLiftOperation | Complete | 1 | | CD\_WEQP\_GAS\_LIFT\_VALVE | cd.CdWeqpGasLiftValve | Complete | 3 | | CD\_WEQP\_GL\_VALVE\_INSPECTION | cd.CdWeqpGlValveInspection | Complete | 1 | | CD\_WEQP\_GL\_VALVE\_OPERATION | cd.CdWeqpGlValveOperation | Complete | 1 | | CD\_WEQP\_GRAVEL\_PACK\_SCREEN | cd.CdWeqpGravelPackScreen | Complete | 0 | | CD\_WEQP\_PACKER | cd.CdWeqpPacker | Complete | 0 | | CD\_WEQP\_PBR | cd.CdWeqpPbr | Complete | 0 | | CD\_WEQP\_PCP\_DETAIL | cd.CdWeqpPcpDetail | Complete | 1 | | CD\_WEQP\_PD\_SENSOR | cd.CdWeqpPdSensor | Complete | 0 | | CD\_WEQP\_PROGRESSIVE\_CAVITY | cd.CdWeqpProgressiveCavity | Complete | 0 | | CD\_WEQP\_ROD | cd.CdWeqpRod | Complete | 0 | | CD\_WEQP\_SSSV | cd.CdWeqpSssv | Complete | 0 | | CD\_WEQP\_TIEBACK\_RECEPT | cd.CdWeqpTiebackRecept | Complete | 0 | | CWO\_OPERATION\_HEADER | cw\.CwoOperationHeader | Complete | 1 | | CW\_DATA\_MODEL | cw\.CwDataModel | Complete | 2 | | CW\_DATA\_MODEL\_HEADER | cw\.CwDataModelHeader | Complete | 1 | | CW\_DATA\_MODEL\_POINTS | cw\.CwDataModelPoints | Complete | 1 | | CW\_DOGLEG\_SEVERITY\_OVERRIDE | cw\.CwDoglegSeverityOverride | Complete | 0 | | CW\_GLOBAL\_WF\_CASING | cw\.CwGlobalWfCasing | Complete | 0 | | CW\_GLOBAL\_WF\_STRING | cw\.CwGlobalWfString | Complete | 0 | | CW\_OPERATION | cw\.CwOperation | Complete | 3 | | CW\_OPERATION\_WF\_CASING | cw\.CwOperationWfCasing | Complete | 1 | | CW\_OPERATION\_WF\_STRING | cw\.CwOperationWfString | Complete | 1 | | CW\_PARAMS | cw\.CwParams | Complete | 0 | | CW\_WEAR\_DATA\_HEADER | cw\.CwWearDataHeader | Complete | 0 | | CW\_WEAR\_DATA\_SOURCE\_CASING | cw\.CwWearDataSourceCasing | Complete | 0 | | CW\_WEAR\_FACTOR | cw\.CwWearFactor | Complete | 1 | | CW\_WEAR\_FACTOR\_CATALOG | cw\.CwWearFactorCatalog | Complete | 1 | | CW\_WEAR\_LOG | cw\.CwWearLog | Complete | 1 | | CW\_WEAR\_LOG\_HEADER | cw\.CwWearLogHeader | Complete | 1 | | DA\_COL | da.DaCol | Complete | 0 | | DA\_FILT | da.DaFilt | Complete | 0 | | DA\_GRAPH | da.DaGraph | Complete | 0 | | DA\_QUERY | da.DaQuery | Complete | 0 | | DA\_TMPL | da.DaTmpl | Complete | 0 | | DA\_TREE\_DATA\_ELEMENT | da.DaTreeDataElement | Complete | 0 | | DA\_TREE\_STRUCTURE | da.DaTreeStructure | Complete | 0 | | DA\_TREE\_TABLE\_TYPE | da.DaTreeTableType | Complete | 0 | | DA\_TREE\_VIEW | da.DaTreeView | Complete | 0 | | DA\_T\_Q | da.DaTQ | Complete | 0 | | DM\_ACTIVITY | dm.DmActivity | Complete | 1 | | DM\_ACTIVITY\_DETAIL | dm.DmActivityDetail | Complete | 1 | | DM\_ACTIVITY\_DETAIL\_HEADER | dm.DmActivityDetailHeader | Complete | 1 | | DM\_AFE\_EVENT\_LINK | dm.DmAfeEventLink | Complete | 0 | | DM\_AFE\_REMARKS | dm.DmAfeRemarks | Complete | 0 | | DM\_AFE\_SUPP | dm.DmAfeSupp | Complete | 0 | | DM\_AFE\_WELLBORE\_LINK | dm.DmAfeWellboreLink | Complete | 0 | | DM\_AFE\_WELL\_LINK | dm.DmAfeWellLink | Complete | 0 | | DM\_ANCHOR\_OP | dm.DmAnchorOp | Complete | 1 | | DM\_ASSEMBLY\_COMP\_OP | dm.DmAssemblyCompOp | Complete | 0 | | DM\_BHA\_OP | dm.DmBhaOp | Complete | 1 | | DM\_BHA\_SENSOR | dm.DmBhaSensor | Complete | 0 | | DM\_BIT\_OP | dm.DmBitOp | Complete | 1 | | DM\_BOILER\_OP | dm.DmBoilerOp | Complete | 1 | | DM\_BULK | dm.DmBulk | Complete | 0 | | DM\_BULK\_TRAN | dm.DmBulkTran | Complete | 1 | | DM\_CARBIDE\_LAG | dm.DmCarbideLag | Complete | 1 | | DM\_CASING\_FLANGE | dm.DmCasingFlange | Complete | 0 | | DM\_CEMENT\_INTERMEDIATE\_CIRC | dm.DmCementIntermediateCirc | Complete | 1 | | DM\_CEMENT\_PUMP\_EQUIPMENT | dm.DmCementPumpEquipment | Complete | 1 | | DM\_CEMENT\_VG\_METER | dm.DmCementVgMeter | Complete | 1 | | DM\_CENTRIFUGE\_OP | dm.DmCentrifugeOp | Complete | 1 | | DM\_CHROMATOGRAPHY | dm.DmChromatography | Complete | 1 | | DM\_COMPANY | dm.DmCompany | Complete | 0 | | DM\_COMPANY\_DAILY | dm.DmCompanyDaily | Complete | 1 | | DM\_CONV\_PUMP | dm.DmConvPump | Complete | 3 | | DM\_CORE | dm.DmCore | Complete | 1 | | DM\_CORE\_TIME | dm.DmCoreTime | Complete | 1 | | DM\_COUNTER\_WEIGHT | dm.DmCounterWeight | Complete | 1 | | DM\_CREW | dm.DmCrew | Complete | 0 | | DM\_CREW\_OP | dm.DmCrewOp | Complete | 1 | | DM\_CR\_ABANDON\_CONTRACTOR | dm.DmCrAbandonContractor | Complete | 1 | | DM\_CR\_ACCESS\_CONTR | dm.DmCrAccessContr | Complete | 0 | | DM\_CR\_ACC\_ANOMALY | dm.DmCrAccAnomaly | Complete | 0 | | DM\_CR\_AC\_SOIL\_COND | dm.DmCrAcSoilCond | Complete | 0 | | DM\_CR\_AM\_FERTILIZER | dm.DmCrAmFertilizer | Complete | 1 | | DM\_CR\_AM\_SEEDING | dm.DmCrAmSeeding | Complete | 2 | | DM\_CR\_AM\_SEED\_MIX | dm.DmCrAmSeedMix | Complete | 1 | | DM\_CR\_AM\_SOIL\_AMEND | dm.DmCrAmSoilAmend | Complete | 1 | | DM\_CR\_AM\_SOIL\_COND | dm.DmCrAmSoilCond | Complete | 1 | | DM\_CR\_AM\_SOIL\_TREAT | dm.DmCrAmSoilTreat | Complete | 1 | | DM\_CR\_ANOMALY | dm.DmCrAnomaly | Complete | 0 | | DM\_CR\_APPL | dm.DmCrAppl | Complete | 1 | | DM\_CR\_APPROVAL | dm.DmCrApproval | Complete | 1 | | DM\_CR\_CLEANUP | dm.DmCrCleanup | Complete | 11 | | DM\_CR\_CONTROL\_VEG | dm.DmCrControlVeg | Complete | 1 | | DM\_CR\_CORRESPONDENCE | dm.DmCrCorrespondence | Complete | 1 | | DM\_CR\_DECOM\_APPROVAL | dm.DmCrDecomApproval | Complete | 1 | | DM\_CR\_DECOM\_CONTRACTOR | dm.DmCrDecomContractor | Complete | 1 | | DM\_CR\_DECOM\_FACILITY | dm.DmCrDecomFacility | Complete | 1 | | DM\_CR\_DECOM\_PIPELINE | dm.DmCrDecomPipeline | Complete | 1 | | DM\_CR\_DRILLING\_MUD | dm.DmCrDrillingMud | Complete | 1 | | DM\_CR\_GRAVEL\_CONTR | dm.DmCrGravelContr | Complete | 0 | | DM\_CR\_GRAVEL\_USED | dm.DmCrGravelUsed | Complete | 0 | | DM\_CR\_LANDOWNER\_COMMENTS | dm.DmCrLandownerComments | Complete | 0 | | DM\_CR\_LANDOWNER\_OCCUPANT | dm.DmCrLandownerOccupant | Complete | 0 | | DM\_CR\_LAND\_USE | dm.DmCrLandUse | Complete | 0 | | DM\_CR\_MUD\_ADDITIVE | dm.DmCrMudAdditive | Complete | 1 | | DM\_CR\_MUD\_DISPOSAL | dm.DmCrMudDisposal | Complete | 1 | | DM\_CR\_OPER\_CONTACT | dm.DmCrOperContact | Complete | 1 | | DM\_CR\_PIPELINE\_SEG | dm.DmCrPipelineSeg | Complete | 1 | | DM\_CR\_PL\_SEG\_STATUS | dm.DmCrPlSegStatus | Complete | 1 | | DM\_CR\_RECLAMATION | dm.DmCrReclamation | Complete | 24 | | DM\_CR\_RECL\_AGST | dm.DmCrReclAgst | Complete | 1 | | DM\_CR\_RECL\_FLARE\_PIT | dm.DmCrReclFlarePit | Complete | 1 | | DM\_CR\_RECL\_UST | dm.DmCrReclUst | Complete | 1 | | DM\_CR\_REMED\_CONTRACTOR | dm.DmCrRemedContractor | Complete | 1 | | DM\_CR\_REMED\_CRITERIA | dm.DmCrRemedCriteria | Complete | 1 | | DM\_CR\_RESTORE\_CONTR | dm.DmCrRestoreContr | Complete | 1 | | DM\_CR\_RST\_ASSESSMENT | dm.DmCrRstAssessment | Complete | 1 | | DM\_CR\_RST\_FERTILIZER | dm.DmCrRstFertilizer | Complete | 1 | | DM\_CR\_RST\_SEEDING | dm.DmCrRstSeeding | Complete | 2 | | DM\_CR\_RST\_SEED\_MIX | dm.DmCrRstSeedMix | Complete | 1 | | DM\_CR\_RST\_SOIL\_AMEND | dm.DmCrRstSoilAmend | Complete | 1 | | DM\_CR\_SALVAGE\_CONTRACT | dm.DmCrSalvageContract | Complete | 1 | | DM\_CR\_SCOUT\_ASSESS | dm.DmCrScoutAssess | Complete | 0 | | DM\_CR\_SCOUT\_STAKEHOLDER | dm.DmCrScoutStakeholder | Complete | 0 | | DM\_CR\_SCOUT\_SUPP | dm.DmCrScoutSupp | Complete | 0 | | DM\_CR\_SC\_AC\_SOIL\_COND | dm.DmCrScAcSoilCond | Complete | 0 | | DM\_CR\_SC\_DISPOSITION | dm.DmCrScDisposition | Complete | 0 | | DM\_CR\_SC\_VEGETATION | dm.DmCrScVegetation | Complete | 0 | | DM\_CR\_SITE\_VEGETATION | dm.DmCrSiteVegetation | Complete | 1 | | DM\_CR\_SOIL\_CONDITION | dm.DmCrSoilCondition | Complete | 0 | | DM\_CR\_SPILL | dm.DmCrSpill | Complete | 1 | | DM\_CR\_STAKEHOLDER | dm.DmCrStakeholder | Complete | 1 | | DM\_CR\_SUMP | dm.DmCrSump | Complete | 0 | | DM\_CR\_SUMP\_DISPOSITION | dm.DmCrSumpDisposition | Complete | 1 | | DM\_CR\_SUMP\_SAMPLE | dm.DmCrSumpSample | Complete | 1 | | DM\_CR\_TFA | dm.DmCrTfa | Complete | 0 | | DM\_CR\_UTILITY | dm.DmCrUtility | Complete | 1 | | DM\_CR\_UTILITY\_CROSSING | dm.DmCrUtilityCrossing | Complete | 1 | | DM\_CR\_WATER\_WELL | dm.DmCrWaterWell | Complete | 1 | | DM\_CR\_WATER\_WELL\_ABANDON | dm.DmCrWaterWellAbandon | Complete | 1 | | DM\_CR\_WATER\_WELL\_TEST | dm.DmCrWaterWellTest | Complete | 1 | | DM\_CR\_WST\_ANOMALY | dm.DmCrWstAnomaly | Complete | 0 | | DM\_CR\_WS\_SUBSOIL | dm.DmCrWsSubsoil | Complete | 0 | | DM\_CR\_WS\_TOP\_SOIL | dm.DmCrWsTopSoil | Complete | 0 | | DM\_CUTTINGS | dm.DmCuttings | Complete | 1 | | DM\_DAILY | dm.DmDaily | Complete | 49 | | DM\_DAILY\_COMPLETION | dm.DmDailyCompletion | Complete | 1 | | DM\_DAILY\_CT\_OP | dm.DmDailyCtOp | Complete | 1 | | DM\_DAILY\_DITCH\_MAGNET | dm.DmDailyDitchMagnet | Complete | 1 | | DM\_DAILY\_EQUIPMENT | dm.DmDailyEquipment | Complete | 1 | | DM\_DAILY\_FLARING | dm.DmDailyFlaring | Complete | 1 | | DM\_DAILY\_GLV\_TEST | dm.DmDailyGlvTest | Complete | 1 | | DM\_DAILY\_NOTIFICATION | dm.DmDailyNotification | Complete | 1 | | DM\_DAILY\_SSSV\_TEST | dm.DmDailySssvTest | Complete | 1 | | DM\_DAILY\_UBD | dm.DmDailyUbd | Complete | 1 | | DM\_DEGASSER\_OP | dm.DmDegasserOp | Complete | 1 | | DM\_DISCHARGE | dm.DmDischarge | Complete | 1 | | DM\_DISPOSAL | dm.DmDisposal | Complete | 1 | | DM\_DRILLINE | dm.DmDrilline | Complete | 0 | | DM\_DRILLINE\_OP | dm.DmDrillineOp | Complete | 1 | | DM\_DRILL\_METHOD | dm.DmDrillMethod | Complete | 0 | | DM\_DRILL\_TECHNIQUE | dm.DmDrillTechnique | Complete | 0 | | DM\_DST | dm.DmDst | Complete | 6 | | DM\_DST\_ANALYSIS\_LINK | dm.DmDstAnalysisLink | Complete | 2 | | DM\_DST\_CC\_DATA | dm.DmDstCcData | Complete | 1 | | DM\_DST\_CC\_FLOW | dm.DmDstCcFlow | Complete | 2 | | DM\_DST\_FLOW | dm.DmDstFlow | Complete | 2 | | DM\_DST\_FLOW\_RECORDER | dm.DmDstFlowRecorder | Complete | 1 | | DM\_DST\_RATE | dm.DmDstRate | Complete | 1 | | DM\_DST\_RECORDER | dm.DmDstRecorder | Complete | 3 | | DM\_DST\_RECORDER\_DATA | dm.DmDstRecorderData | Complete | 1 | | DM\_DST\_RECOV | dm.DmDstRecov | Complete | 1 | | DM\_DST\_SAMPLE | dm.DmDstSample | Complete | 1 | | DM\_ENVIRONMENTAL | dm.DmEnvironmental | Complete | 1 | | DM\_ESP | dm.DmEsp | Complete | 13 | | DM\_ESP\_CABLE\_INSPECTION | dm.DmEspCableInspection | Complete | 2 | | DM\_ESP\_CABLE\_INSPECTION\_DET | dm.DmEspCableInspectionDet | Complete | 1 | | DM\_ESP\_COMMISSION | dm.DmEspCommission | Complete | 1 | | DM\_ESP\_COMPLEMENTARY\_DATA | dm.DmEspComplementaryData | Complete | 1 | | DM\_ESP\_COMPONENT | dm.DmEspComponent | Complete | 1 | | DM\_ESP\_DESIGN | dm.DmEspDesign | Complete | 1 | | DM\_ESP\_DETAILS | dm.DmEspDetails | Complete | 1 | | DM\_ESP\_ELECTRIC\_DETAIL | dm.DmEspElectricDetail | Complete | 1 | | DM\_ESP\_ELECTRIC\_INSPECTION | dm.DmEspElectricInspection | Complete | 1 | | DM\_ESP\_MAIN\_EVENTS | dm.DmEspMainEvents | Complete | 1 | | DM\_ESP\_OPERATIONAL\_COND | dm.DmEspOperationalCond | Complete | 1 | | DM\_ESP\_POWER | dm.DmEspPower | Complete | 1 | | DM\_ESP\_PULL | dm.DmEspPull | Complete | 1 | | DM\_ESP\_WELL\_AREA\_INSPECTION | dm.DmEspWellAreaInspection | Complete | 1 | | DM\_EVENT\_COMPLETION\_LINK | dm.DmEventCompletionLink | Complete | 0 | | DM\_EVENT\_CONTRACT\_LINK | dm.DmEventContractLink | Complete | 0 | | DM\_EVENT\_COST | dm.DmEventCost | Complete | 0 | | DM\_FAILURE\_CASING | dm.DmFailureCasing | Complete | 1 | | DM\_FAILURE\_CASING\_STRING | dm.DmFailureCasingString | Complete | 1 | | DM\_FAILURE\_CLEANOUT | dm.DmFailureCleanout | Complete | 1 | | DM\_FAILURE\_CONV\_PUMP | dm.DmFailureConvPump | Complete | 1 | | DM\_FAILURE\_COST | dm.DmFailureCost | Complete | 1 | | DM\_FAILURE\_ESP | dm.DmFailureEsp | Complete | 1 | | DM\_FAILURE\_GAS\_LIFT | dm.DmFailureGasLift | Complete | 1 | | DM\_FAILURE\_PCP | dm.DmFailurePcp | Complete | 1 | | DM\_FAILURE\_ROD | dm.DmFailureRod | Complete | 1 | | DM\_FAILURE\_ROD\_BREAK | dm.DmFailureRodBreak | Complete | 1 | | DM\_FAILURE\_TUBING\_LEAK | dm.DmFailureTubingLeak | Complete | 1 | | DM\_FAILURE\_TUBING\_STRING | dm.DmFailureTubingString | Complete | 1 | | DM\_FAILURE\_WELLBORE\_EQUIP | dm.DmFailureWellboreEquip | Complete | 1 | | DM\_FISHING | dm.DmFishing | Complete | 4 | | DM\_FISHING\_FREE\_POINT | dm.DmFishingFreePoint | Complete | 1 | | DM\_FISHING\_RESPONSIBLE | dm.DmFishingResponsible | Complete | 1 | | DM\_FISHING\_RESULTS | dm.DmFishingResults | Complete | 1 | | DM\_FISHING\_STRING\_SHOT | dm.DmFishingStringShot | Complete | 1 | | DM\_FLUID\_BALANCE | dm.DmFluidBalance | Complete | 2 | | DM\_FLUID\_HAUL | dm.DmFluidHaul | Complete | 1 | | DM\_FLUID\_HAUL\_DETAIL | dm.DmFluidHaulDetail | Complete | 1 | | DM\_FLUID\_INPUT | dm.DmFluidInput | Complete | 1 | | DM\_FLUID\_LOSS | dm.DmFluidLoss | Complete | 2 | | DM\_FLUID\_LOSS\_HOLE\_SECT | dm.DmFluidLossHoleSect | Complete | 1 | | DM\_GAS\_LIFT | dm.DmGasLift | Complete | 1 | | DM\_GAS\_PEAK | dm.DmGasPeak | Complete | 1 | | DM\_GENERAL\_WORK | dm.DmGeneralWork | Complete | 1 | | DM\_GEOLOGY\_DAILY | dm.DmGeologyDaily | Complete | 4 | | DM\_GRAVEL\_PACK | dm.DmGravelPack | Complete | 4 | | DM\_GRAVEL\_PACK\_PERF\_INTERVAL | dm.DmGravelPackPerfInterval | Complete | 1 | | DM\_GRAVEL\_PACK\_PUMP\_SCHEDULE | dm.DmGravelPackPumpSchedule | Complete | 1 | | DM\_GRAVEL\_PACK\_SAND | dm.DmGravelPackSand | Complete | 1 | | DM\_GRAVEL\_PACK\_WASHOUT | dm.DmGravelPackWashout | Complete | 1 | | DM\_GUIDELINE | dm.DmGuideline | Complete | 0 | | DM\_GUIDELINE\_OP | dm.DmGuidelineOp | Complete | 1 | | DM\_HSE | dm.DmHse | Complete | 1 | | DM\_HYDROCLONE\_OP | dm.DmHydrocloneOp | Complete | 1 | | DM\_INCIDENT\_COST | dm.DmIncidentCost | Complete | 0 | | DM\_INCIDENT\_DETAIL | dm.DmIncidentDetail | Complete | 0 | | DM\_INSPECTION | dm.DmInspection | Complete | 2 | | DM\_INTERMEDIATE\_CIRC | dm.DmIntermediateCirc | Complete | 0 | | DM\_INTERVAL | dm.DmInterval | Complete | 3 | | DM\_INTERVAL\_LITHOLOGY | dm.DmIntervalLithology | Complete | 1 | | DM\_ISOLATED\_INTERVAL | dm.DmIsolatedInterval | Complete | 1 | | DM\_KICK | dm.DmKick | Complete | 0 | | DM\_KO\_WINDOW | dm.DmKoWindow | Complete | 0 | | DM\_LOG | dm.DmLog | Complete | 4 | | DM\_LOG\_CASING\_COLLAR | dm.DmLogCasingCollar | Complete | 1 | | DM\_LOG\_DESC | dm.DmLogDesc | Complete | 1 | | DM\_LOG\_FORM\_TEST | dm.DmLogFormTest | Complete | 1 | | DM\_LOG\_INTERVAL | dm.DmLogInterval | Complete | 3 | | DM\_LOG\_INTERVAL\_HUD | dm.DmLogIntervalHud | Complete | 1 | | DM\_LOG\_SERVICE | dm.DmLogService | Complete | 1 | | DM\_MATERIAL\_TRANSFER | dm.DmMaterialTransfer | Complete | 1 | | DM\_MOTOR\_OP | dm.DmMotorOp | Complete | 1 | | DM\_MOVE\_OP | dm.DmMoveOp | Complete | 0 | | DM\_MOVE\_PRELOAD | dm.DmMovePreload | Complete | 0 | | DM\_MOVE\_VESSEL | dm.DmMoveVessel | Complete | 0 | | DM\_MUDGAS | dm.DmMudgas | Complete | 1 | | DM\_MUD\_PRODUCT | dm.DmMudProduct | Complete | 0 | | DM\_MUD\_PRODUCT\_TRAN | dm.DmMudProductTran | Complete | 1 | | DM\_MUD\_VOLUME | dm.DmMudVolume | Complete | 1 | | DM\_OPER\_EQUIP\_FAIL\_STATUS | dm.DmOperEquipFailStatus | Complete | 0 | | DM\_PARTNER | dm.DmPartner | Complete | 1 | | DM\_PARTNER\_COMPLETION | dm.DmPartnerCompletion | Complete | 0 | | DM\_PARTNER\_CONTACT | dm.DmPartnerContact | Complete | 2 | | DM\_PARTNER\_CONTACT\_EVENTCODE | dm.DmPartnerContactEventcode | Complete | 1 | | DM\_PARTNER\_EVENT | dm.DmPartnerEvent | Complete | 0 | | DM\_PARTNER\_WELL | dm.DmPartnerWell | Complete | 0 | | DM\_PARTNER\_WELLBORE | dm.DmPartnerWellbore | Complete | 0 | | DM\_PCP | dm.DmPcp | Complete | 5 | | DM\_PCP\_COMPONENT | dm.DmPcpComponent | Complete | 1 | | DM\_PCP\_FLUID | dm.DmPcpFluid | Complete | 1 | | DM\_PCP\_OPERATION | dm.DmPcpOperation | Complete | 1 | | DM\_PCP\_STATUS\_HISTORY | dm.DmPcpStatusHistory | Complete | 1 | | DM\_PERSONNEL | dm.DmPersonnel | Complete | 0 | | DM\_PERSONNEL\_DAILY | dm.DmPersonnelDaily | Complete | 1 | | DM\_PIPE\_DATA | dm.DmPipeData | Complete | 1 | | DM\_PIPE\_RUN | dm.DmPipeRun | Complete | 2 | | DM\_PIPE\_TALLY | dm.DmPipeTally | Complete | 1 | | DM\_PIT\_OP | dm.DmPitOp | Complete | 1 | | DM\_PLAN\_DRILL\_PARAM | dm.DmPlanDrillParam | Complete | 0 | | DM\_PLUNGER\_ACC | dm.DmPlungerAcc | Complete | 1 | | DM\_PLUNGER\_LIFT | dm.DmPlungerLift | Complete | 2 | | DM\_PLUNGER\_LIFT\_STAGE\_SYSTEM | dm.DmPlungerLiftStageSystem | Complete | 1 | | DM\_PRIME\_MOVER | dm.DmPrimeMover | Complete | 1 | | DM\_PRODUCED | dm.DmProduced | Complete | 1 | | DM\_PROD\_EQUIP\_FAIL | dm.DmProdEquipFail | Complete | 15 | | DM\_PROD\_EQUIP\_FAIL\_STATUS | dm.DmProdEquipFailStatus | Complete | 1 | | DM\_PUMP\_JACK | dm.DmPumpJack | Complete | 3 | | DM\_PUMP\_OP | dm.DmPumpOp | Complete | 1 | | DM\_RECOVERY | dm.DmRecovery | Complete | 1 | | DM\_REPORT\_COMMENTS | dm.DmReportComments | Complete | 0 | | DM\_RIG\_ACTIVITY | dm.DmRigActivity | Complete | 1 | | DM\_RIG\_CONTRACT | dm.DmRigContract | Complete | 0 | | DM\_RIG\_DECKLOG | dm.DmRigDecklog | Complete | 2 | | DM\_RIG\_DECKLOG\_REMARKS | dm.DmRigDecklogRemarks | Complete | 1 | | DM\_RIG\_OPERATION\_EVENT\_LINK | dm.DmRigOperationEventLink | Complete | 0 | | DM\_RISERLINE | dm.DmRiserline | Complete | 0 | | DM\_RISER\_OP | dm.DmRiserOp | Complete | 1 | | DM\_RJ\_INTERVAL\_LINK | dm.DmRjIntervalLink | Complete | 1 | | DM\_RUSHMORE\_DPR | dm.DmRushmoreDpr | Complete | 3 | | DM\_RUSHMORE\_DPR\_TD | dm.DmRushmoreDprTd | Complete | 1 | | DM\_RUSHMORE\_DPR\_WELLPATH | dm.DmRushmoreDprWellpath | Complete | 1 | | DM\_RUSHMORE\_SUBMISSION | dm.DmRushmoreSubmission | Complete | 1 | | DM\_SAFETY | dm.DmSafety | Complete | 2 | | DM\_SAFETY\_INCIDENT | dm.DmSafetyIncident | Complete | 1 | | DM\_SAFETY\_KICK\_DETECT | dm.DmSafetyKickDetect | Complete | 1 | | DM\_SHAKERSCREEN | dm.DmShakerscreen | Complete | 0 | | DM\_SHAKERSCREEN\_OP | dm.DmShakerscreenOp | Complete | 1 | | DM\_SHAKER\_OP | dm.DmShakerOp | Complete | 1 | | DM\_SHOW | dm.DmShow | Complete | 1 | | DM\_SLICKLINE | dm.DmSlickline | Complete | 1 | | DM\_SLICKLINE\_DAILY | dm.DmSlicklineDaily | Complete | 2 | | DM\_SLICKLINE\_PURPOSE | dm.DmSlicklinePurpose | Complete | 1 | | DM\_SLICKLINE\_SOLID | dm.DmSlicklineSolid | Complete | 1 | | DM\_SOLID\_DETAIL | dm.DmSolidDetail | Complete | 1 | | DM\_SPILLS | dm.DmSpills | Complete | 1 | | DM\_STIM\_FET\_TEST | dm.DmStimFetTest | Complete | 1 | | DM\_STIM\_FLOWBACK | dm.DmStimFlowback | Complete | 1 | | DM\_STIM\_FLOWPATH | dm.DmStimFlowpath | Complete | 2 | | DM\_STIM\_FLOWPATH\_STAGE | dm.DmStimFlowpathStage | Complete | 1 | | DM\_STIM\_FLUID | dm.DmStimFluid | Complete | 2 | | DM\_STIM\_FLUID\_ADD | dm.DmStimFluidAdd | Complete | 1 | | DM\_STIM\_FLUID\_SCHEDULE | dm.DmStimFluidSchedule | Complete | 1 | | DM\_STIM\_JOB | dm.DmStimJob | Complete | 3 | | DM\_STIM\_PUMPING\_DIAG | dm.DmStimPumpingDiag | Complete | 4 | | DM\_STIM\_PUMP\_FB\_TEST | dm.DmStimPumpFbTest | Complete | 1 | | DM\_STIM\_RES\_INTERVAL | dm.DmStimResInterval | Complete | 1 | | DM\_STIM\_STAGE | dm.DmStimStage | Complete | 3 | | DM\_STIM\_STAGE\_SUPPLEMENT | dm.DmStimStageSupplement | Complete | 1 | | DM\_STIM\_STEP\_TEST | dm.DmStimStepTest | Complete | 1 | | DM\_STIM\_TREATMENT | dm.DmStimTreatment | Complete | 8 | | DM\_STIM\_TREATMENT\_AC\_COMP | dm.DmStimTreatmentAcComp | Complete | 1 | | DM\_STIM\_TREATMENT\_ADDITIVE | dm.DmStimTreatmentAdditive | Complete | 2 | | DM\_STIM\_TREATMENT\_FLUID | dm.DmStimTreatmentFluid | Complete | 1 | | DM\_STIM\_TREATMENT\_PROPPANT | dm.DmStimTreatmentProppant | Complete | 1 | | DM\_SUPPORT\_VESSEL | dm.DmSupportVessel | Complete | 0 | | DM\_SUPPORT\_VESSEL\_DAILY | dm.DmSupportVesselDaily | Complete | 1 | | DM\_TRANSFER\_DETAIL | dm.DmTransferDetail | Complete | 1 | | DM\_TRANSFER\_SUMMARY | dm.DmTransferSummary | Complete | 2 | | DM\_UNDERWATER\_DETAIL | dm.DmUnderwaterDetail | Complete | 1 | | DM\_UNDERWATER\_EQUIPMENT | dm.DmUnderwaterEquipment | Complete | 1 | | DM\_UNDERWATER\_MATERIAL | dm.DmUnderwaterMaterial | Complete | 0 | | DM\_UNDERWATER\_MATERIAL\_TRAN | dm.DmUnderwaterMaterialTran | Complete | 1 | | DM\_UNDERWATER\_OP | dm.DmUnderwaterOp | Complete | 4 | | DM\_UNDERWATER\_PERS | dm.DmUnderwaterPers | Complete | 1 | | DM\_VESSEL\_BULK\_TRAN | dm.DmVesselBulkTran | Complete | 1 | | DM\_WASTE\_TRANSFER\_DETAIL | dm.DmWasteTransferDetail | Complete | 0 | | DM\_WB\_OBSTRUCTION\_CONTENT | dm.DmWbObstructionContent | Complete | 1 | | DM\_WB\_OBSTRUCTION\_STATUS | dm.DmWbObstructionStatus | Complete | 2 | | DM\_WEATHER\_CHECK | dm.DmWeatherCheck | Complete | 2 | | DM\_WEATHER\_CURRENT | dm.DmWeatherCurrent | Complete | 1 | | DM\_WELLBORE\_INTEG | dm.DmWellboreInteg | Complete | 1 | | DM\_WELLBORE\_INTEG\_TEST | dm.DmWellboreIntegTest | Complete | 1 | | DM\_WELLBORE\_LIST | dm.DmWellboreList | Complete | 1 | | DM\_WELLBORE\_LIST\_DETAIL | dm.DmWellboreListDetail | Complete | 1 | | DM\_WELLBORE\_OBSTRUCTION | dm.DmWellboreObstruction | Complete | 1 | | DM\_WELLBORE\_ZONE\_ACTIVITY | dm.DmWellboreZoneActivity | Complete | 0 | | DM\_WELL\_CONTRACT\_LINK | dm.DmWellContractLink | Complete | 0 | | DM\_WELL\_PLAN | dm.DmWellPlan | Complete | 18 | | DM\_WELL\_PLAN\_BIT | dm.DmWellPlanBit | Complete | 1 | | DM\_WELL\_PLAN\_CASING | dm.DmWellPlanCasing | Complete | 2 | | DM\_WELL\_PLAN\_CASING\_TEST | dm.DmWellPlanCasingTest | Complete | 1 | | DM\_WELL\_PLAN\_CEMENT | dm.DmWellPlanCement | Complete | 1 | | DM\_WELL\_PLAN\_CORING | dm.DmWellPlanCoring | Complete | 1 | | DM\_WELL\_PLAN\_DRILLSTRING | dm.DmWellPlanDrillstring | Complete | 1 | | DM\_WELL\_PLAN\_EQUIP | dm.DmWellPlanEquip | Complete | 1 | | DM\_WELL\_PLAN\_GUIDELINE | dm.DmWellPlanGuideline | Complete | 1 | | DM\_WELL\_PLAN\_HAZARD | dm.DmWellPlanHazard | Complete | 1 | | DM\_WELL\_PLAN\_HOLE\_SECT | dm.DmWellPlanHoleSect | Complete | 2 | | DM\_WELL\_PLAN\_HSE | dm.DmWellPlanHse | Complete | 1 | | DM\_WELL\_PLAN\_LOGGING | dm.DmWellPlanLogging | Complete | 1 | | DM\_WELL\_PLAN\_MUD | dm.DmWellPlanMud | Complete | 2 | | DM\_WELL\_PLAN\_MUD\_CLEANING | dm.DmWellPlanMudCleaning | Complete | 1 | | DM\_WELL\_PLAN\_OFFSET\_SUMMARY | dm.DmWellPlanOffsetSummary | Complete | 0 | | DM\_WELL\_PLAN\_OP | dm.DmWellPlanOp | Complete | 1 | | DM\_WELL\_PLAN\_PROD\_ACTUAL | dm.DmWellPlanProdActual | Complete | 1 | | DM\_WELL\_PLAN\_PROD\_BASE | dm.DmWellPlanProdBase | Complete | 1 | | DM\_WELL\_PLAN\_PROD\_PLAN | dm.DmWellPlanProdPlan | Complete | 1 | | DM\_WELL\_PLAN\_TEST | dm.DmWellPlanTest | Complete | 1 | | DM\_WELL\_PLAN\_TREATMENT | dm.DmWellPlanTreatment | Complete | 1 | | DM\_WELL\_PLAN\_WELLBORE\_LINK | dm.DmWellPlanWellboreLink | Complete | 1 | | DM\_WORK\_DETAIL | dm.DmWorkDetail | Complete | 1 | | DP\_ACRULE | dp.DpAcrule | Complete | 0 | | DP\_ANNOTATION | dp.DpAnnotation | Complete | 0 | | DP\_ANTICOL | dp.DpAnticol | Complete | 0 | | DP\_DEF\_SURVEY\_ANNOTATION | dp.DpDefSurveyAnnotation | Complete | 0 | | DP\_MAGNETIC | dp.DpMagnetic | Complete | 0 | | DP\_PROJECT\_TARGET\_POINT | dp.DpProjectTargetPoint | Complete | 0 | | DP\_SURVEY\_STN\_SENT\_ALERTS | dp.DpSurveyStnSentAlerts | Complete | 0 | | DP\_TCOLOR | dp.DpTcolor | Complete | 0 | | DP\_TOOL\_TERM | dp.DpToolTerm | Complete | 0 | | DP\_WALK | dp.DpWalk | Complete | 0 | | DS\_APP\_FILE | ds.DsAppFile | Complete | 0 | | DT\_APPLICATION | dt.DtApplication | Complete | 0 | | DT\_DP\_FIELD | dt.DtDpField | Complete | 0 | | DT\_MAP | dt.DtMap | Complete | 0 | | DT\_MAPPING\_SET | dt.DtMappingSet | Complete | 0 | | DT\_MAP\_TABLE | dt.DtMapTable | Complete | 1 | | DT\_MAP\_VALUE | dt.DtMapValue | Complete | 1 | | DT\_RESERVED\_WORD | dt.DtReservedWord | Complete | 0 | | DT\_SCRIPT | dt.DtScript | Complete | 0 | | DT\_SESSION | dt.DtSession | Complete | 0 | | DT\_SOURCE\_FIELD | dt.DtSourceField | Complete | 0 | | DT\_STATIC\_KEY | dt.DtStaticKey | Complete | 0 | | DT\_TARGET\_FIELD | dt.DtTargetField | Complete | 0 | | KA\_DEFINE | ka.KaDefine | Complete | 0 | | KA\_LINKS | ka.KaLinks | Complete | 1 | | KA\_OPER\_CONFIG | ka.KaOperConfig | Complete | 0 | | KA\_RESULTS | ka.KaResults | Complete | 1 | | KA\_SYMBOLS | ka.KaSymbols | Complete | 0 | | MD\_CATALOG\_TYPE | md.MdCatalogType | Complete | 0 | | MD\_DB\_GENERIC\_RESOURCE | md.MdDbGenericResource | Complete | 0 | | MD\_DEPTH\_TYPE | md.MdDepthType | Complete | 0 | | MD\_ERROR\_MODEL | md.MdErrorModel | Complete | 0 | | MD\_ERROR\_SURFACE | md.MdErrorSurface | Complete | 0 | | MD\_HIERARCHY\_ENTITY\_BINDING | md.MdHierarchyEntityBinding | Complete | 0 | | MD\_HIERARCHY\_LEVEL\_OBJECTS | md.MdHierarchyLevelObjects | Complete | 1 | | MD\_HIERARCHY\_SCHEMES | md.MdHierarchySchemes | Complete | 1 | | MD\_HIERARCHY\_SCHEME\_LEVELS | md.MdHierarchySchemeLevels | Complete | 2 | | MD\_HIERARCHY\_SCHEME\_TYPES | md.MdHierarchySchemeTypes | Complete | 0 | | MD\_LC\_ORIGIN | md.MdLcOrigin | Complete | 0 | | MD\_NORTH\_TYPE | md.MdNorthType | Complete | 0 | | MD\_PK\_ACRULE\_TYPE | md.MdPkAcruleType | Complete | 0 | | MD\_PK\_AC\_CASING | md.MdPkAcCasing | Complete | 0 | | MD\_PK\_ANTICOL\_MODEL | md.MdPkAnticolModel | Complete | 0 | | MD\_PK\_ASSEMBLY\_PLUNGER\_V | md.MdPkAssemblyPlungerV | Complete | 0 | | MD\_PK\_ASSEMBLY\_STATUS | md.MdPkAssemblyStatus | Complete | 0 | | MD\_PK\_BOOLEAN | md.MdPkBoolean | Complete | 0 | | MD\_PK\_BRINE\_TYPE | md.MdPkBrineType | Complete | 0 | | MD\_PK\_COMPONENT\_STATUS | md.MdPkComponentStatus | Complete | 0 | | MD\_PK\_COMP\_TYPE | md.MdPkCompType | Complete | 1 | | MD\_PK\_CSG\_COMP\_GROUP | md.MdPkCsgCompGroup | Complete | 0 | | MD\_PK\_DATA\_ENTRY\_MODE | md.MdPkDataEntryMode | Complete | 0 | | MD\_PK\_DATUM\_TYPE | md.MdPkDatumType | Complete | 0 | | MD\_PK\_DRILL\_MODE | md.MdPkDrillMode | Complete | 0 | | MD\_PK\_DST\_RECOV\_IND | md.MdPkDstRecovInd | Complete | 0 | | MD\_PK\_DST\_RECOV\_TYPE | md.MdPkDstRecovType | Complete | 0 | | MD\_PK\_FLUID\_DESIGNATION | md.MdPkFluidDesignation | Complete | 0 | | MD\_PK\_FOAM\_TYPE | md.MdPkFoamType | Complete | 0 | | MD\_PK\_GEO\_PROJECT\_TYPE | md.MdPkGeoProjectType | Complete | 0 | | MD\_PK\_HISTORY\_ACTIONS | md.MdPkHistoryActions | Complete | 0 | | MD\_PK\_HISTORY\_ITEMS | md.MdPkHistoryItems | Complete | 0 | | MD\_PK\_IADC\_BEARING\_TYPE | md.MdPkIadcBearingType | Complete | 0 | | MD\_PK\_IADC\_CUTTER\_HYDRAULIC | md.MdPkIadcCutterHydraulic | Complete | 0 | | MD\_PK\_IADC\_CUTTER\_TYPE | md.MdPkIadcCutterType | Complete | 0 | | MD\_PK\_IADC\_FEATURE | md.MdPkIadcFeature | Complete | 0 | | MD\_PK\_INSTRUMENT\_TYPE | md.MdPkInstrumentType | Complete | 0 | | MD\_PK\_JOB\_STATUS | md.MdPkJobStatus | Complete | 0 | | MD\_PK\_LINE\_STYLE | md.MdPkLineStyle | Complete | 0 | | MD\_PK\_LINE\_TYPE | md.MdPkLineType | Complete | 0 | | MD\_PK\_LOCATION\_TYPE\_CODE | md.MdPkLocationTypeCode | Complete | 0 | | MD\_PK\_MAGNETIC\_TYPE | md.MdPkMagneticType | Complete | 0 | | MD\_PK\_MUD\_BASE\_TYPE | md.MdPkMudBaseType | Complete | 0 | | MD\_PK\_MWD\_TOOL | md.MdPkMwdTool | Complete | 0 | | MD\_PK\_OW\_TEMPLATE | md.MdPkOwTemplate | Complete | 0 | | MD\_PK\_PACKER\_RUN\_STRING | md.MdPkPackerRunString | Complete | 0 | | MD\_PK\_PACKER\_SET\_TYPE | md.MdPkPackerSetType | Complete | 0 | | MD\_PK\_PDEN\_TYPE | md.MdPkPdenType | Complete | 0 | | MD\_PK\_PERF\_REVIEW | md.MdPkPerfReview | Complete | 0 | | MD\_PK\_PHASE | md.MdPkPhase | Complete | 0 | | MD\_PK\_PLUNGER\_V | md.MdPkPlungerV | Complete | 0 | | MD\_PK\_PRESSURE\_SURVEY\_TYPE | md.MdPkPressureSurveyType | Complete | 0 | | MD\_PK\_PROD\_FAILURE\_TYPE | md.MdPkProdFailureType | Complete | 0 | | MD\_PK\_QUALITY\_INDEX | md.MdPkQualityIndex | Complete | 0 | | MD\_PK\_REPORT\_STANDARD | md.MdPkReportStandard | Complete | 0 | | MD\_PK\_RHEOLOGY\_MODEL\_TYPE | md.MdPkRheologyModelType | Complete | 0 | | MD\_PK\_SCAN\_METHOD | md.MdPkScanMethod | Complete | 0 | | MD\_PK\_SHAPE\_TYPE | md.MdPkShapeType | Complete | 0 | | MD\_PK\_SITE\_TYPE | md.MdPkSiteType | Complete | 0 | | MD\_PK\_STRING\_TYPE | md.MdPkStringType | Complete | 0 | | MD\_PK\_SURVEY\_CALC | md.MdPkSurveyCalc | Complete | 0 | | MD\_PK\_SURVEY\_TYPE | md.MdPkSurveyType | Complete | 0 | | MD\_PK\_SYSTEM\_DATUM\_TYPE | md.MdPkSystemDatumType | Complete | 0 | | MD\_PK\_TAML\_CLASS | md.MdPkTamlClass | Complete | 0 | | MD\_PK\_THREAD\_TYPE | md.MdPkThreadType | Complete | 0 | | MD\_PK\_TIE\_ON\_TYPE | md.MdPkTieOnType | Complete | 0 | | MD\_PK\_TLD\_SUM\_LEVEL | md.MdPkTldSumLevel | Complete | 0 | | MD\_PK\_TOOL\_TYPE | md.MdPkToolType | Complete | 0 | | MD\_PK\_TP\_TYPE | md.MdPkTpType | Complete | 0 | | MD\_PK\_WALK\_TYPE | md.MdPkWalkType | Complete | 0 | | MD\_PK\_WARNING\_METHOD | md.MdPkWarningMethod | Complete | 0 | | MD\_PK\_WCN\_INFLUX\_TYPES | md.MdPkWcnInfluxTypes | Complete | 0 | | MD\_PK\_WELL\_TEST\_TYPE | md.MdPkWellTestType | Complete | 0 | | MD\_PK\_ZONATION\_SCHEME\_TYPE | md.MdPkZonationSchemeType | Complete | 0 | | MD\_SECT\_TYPE | md.MdSectType | Complete | 3 | | MD\_SITE\_APP\_ACTION | md.MdSiteAppAction | Complete | 0 | | MD\_SITE\_APP\_ACTION\_OWNER | md.MdSiteAppActionOwner | Complete | 0 | | MD\_SITE\_ATTR\_SET\_OVERRIDE | md.MdSiteAttrSetOverride | Complete | 1 | | MD\_SITE\_CARRYOVER | md.MdSiteCarryover | Complete | 1 | | MD\_SITE\_CEMENT\_JOB\_TYPE | md.MdSiteCementJobType | Complete | 0 | | MD\_SITE\_CONFIG\_TRANSFER | md.MdSiteConfigTransfer | Complete | 0 | | MD\_SITE\_COORD\_TYPE | md.MdSiteCoordType | Complete | 0 | | MD\_SITE\_DEX\_ATTRIBUTES | md.MdSiteDexAttributes | Complete | 0 | | MD\_SITE\_DV\_RULES | md.MdSiteDvRules | Complete | 0 | | MD\_SITE\_EVENT\_TYPE | md.MdSiteEventType | Complete | 0 | | MD\_SITE\_JOB\_TYPE | md.MdSiteJobType | Complete | 2 | | MD\_SITE\_JOB\_TYPE\_REPORT | md.MdSiteJobTypeReport | Complete | 1 | | MD\_SITE\_JOB\_TYPE\_TASK | md.MdSiteJobTypeTask | Complete | 1 | | MD\_SITE\_LOOKUP\_FLAGS | md.MdSiteLookupFlags | Complete | 0 | | MD\_SITE\_OVERRIDES | md.MdSiteOverrides | Complete | 0 | | MD\_SITE\_PWD\_POLICY | md.MdSitePwdPolicy | Complete | 0 | | MD\_SITE\_REG\_MAPPING | md.MdSiteRegMapping | Complete | 0 | | MD\_SITE\_REPORT\_TYPE | md.MdSiteReportType | Complete | 0 | | MD\_SITE\_SETTINGS | md.MdSiteSettings | Complete | 0 | | MD\_SITE\_STATUS\_COLOR | md.MdSiteStatusColor | Complete | 0 | | MD\_SITE\_STORAGE\_UNITS | md.MdSiteStorageUnits | Complete | 1 | | MD\_SITE\_TABLE\_DEF | md.MdSiteTableDef | Complete | 1 | | MD\_SITE\_TIGHT\_GROUP | md.MdSiteTightGroup | Complete | 1 | | MD\_SITE\_TIGHT\_GROUP\_MEMBER | md.MdSiteTightGroupMember | Complete | 2 | | MD\_SITE\_USER | md.MdSiteUser | Complete | 2 | | MD\_SITE\_USER\_PWD\_HISTORY | md.MdSiteUserPwdHistory | Complete | 1 | | MD\_SITE\_USER\_TIGHT\_GROUP | md.MdSiteUserTightGroup | Complete | 0 | | MD\_STRING\_SECT\_MAP | md.MdStringSectMap | Complete | 1 | | MD\_SURVEY\_STATION\_TYPE | md.MdSurveyStationType | Complete | 0 | | MD\_SYS\_ALT\_KEY | md.MdSysAltKey | Complete | 1 | | MD\_SYS\_ALT\_KEY\_COLS | md.MdSysAltKeyCols | Complete | 1 | | MD\_SYS\_ATTR\_DICTIONARY | md.MdSysAttrDictionary | Complete | 4 | | MD\_SYS\_ATTR\_RULES | md.MdSysAttrRules | Complete | 2 | | MD\_SYS\_ATTR\_SETTINGS | md.MdSysAttrSettings | Complete | 1 | | MD\_SYS\_BENGAL\_EDM\_MAP | md.MdSysBengalEdmMap | Complete | 0 | | MD\_SYS\_BENGAL\_EDM\_SUPP\_MAP | md.MdSysBengalEdmSuppMap | Complete | 0 | | MD\_SYS\_BENGAL\_EDM\_TABLE\_MAP | md.MdSysBengalEdmTableMap | Complete | 0 | | MD\_SYS\_BUSOBJ | md.MdSysBusobj | Complete | 5 | | MD\_SYS\_BUSOBJ\_LINKS | md.MdSysBusobjLinks | Complete | 1 | | MD\_SYS\_BUSOBJ\_MINOR | md.MdSysBusobjMinor | Complete | 1 | | MD\_SYS\_BUSOBJ\_RULES | md.MdSysBusobjRules | Complete | 2 | | MD\_SYS\_CONSTRAINTS | md.MdSysConstraints | Complete | 0 | | MD\_SYS\_COPY\_FIELD\_EXCEPTIONS | md.MdSysCopyFieldExceptions | Complete | 0 | | MD\_SYS\_COPY\_RULE | md.MdSysCopyRule | Complete | 0 | | MD\_SYS\_DB\_VERSION | md.MdSysDbVersion | Complete | 0 | | MD\_SYS\_DEX\_ATTRIBUTES | md.MdSysDexAttributes | Complete | 0 | | MD\_SYS\_DEX\_OBJECTS | md.MdSysDexObjects | Complete | 0 | | MD\_SYS\_DEX\_TABLES | md.MdSysDexTables | Complete | 0 | | MD\_SYS\_DS\_ASSIGNED\_VALUE | md.MdSysDsAssignedValue | Complete | 0 | | MD\_SYS\_DS\_BUSOBJ | md.MdSysDsBusobj | Complete | 1 | | MD\_SYS\_DS\_BUSOBJ\_GROUP | md.MdSysDsBusobjGroup | Complete | 1 | | MD\_SYS\_DS\_BUSOBJ\_RELATION | md.MdSysDsBusobjRelation | Complete | 0 | | MD\_SYS\_DS\_BUSOBJ\_STRUCT | md.MdSysDsBusobjStruct | Complete | 2 | | MD\_SYS\_DV\_RULES | md.MdSysDvRules | Complete | 0 | | MD\_SYS\_DV\_TEMPLATES | md.MdSysDvTemplates | Complete | 0 | | MD\_SYS\_ECON\_SUBSET | md.MdSysEconSubset | Complete | 1 | | MD\_SYS\_ECON\_SUBSET\_ARCHIVE | md.MdSysEconSubsetArchive | Complete | 0 | | MD\_SYS\_ECON\_SUBSET\_USER | md.MdSysEconSubsetUser | Complete | 1 | | MD\_SYS\_ECON\_USER\_GROUP | md.MdSysEconUserGroup | Complete | 0 | | MD\_SYS\_ECON\_USER\_GROUP\_MEMBER | md.MdSysEconUserGroupMember | Complete | 0 | | MD\_SYS\_FIELD\_DEF | md.MdSysFieldDef | Complete | 3 | | MD\_SYS\_LOCAL\_COORD\_TYPE | md.MdSysLocalCoordType | Complete | 0 | | MD\_SYS\_LOOKUP\_ATTR\_FLAG | md.MdSysLookupAttrFlag | Complete | 0 | | MD\_SYS\_LOOKUP\_ATTR\_RULES | md.MdSysLookupAttrRules | Complete | 1 | | MD\_SYS\_LOOKUP\_ATTR\_TYPE | md.MdSysLookupAttrType | Complete | 0 | | MD\_SYS\_LOOKUP\_BUSOBJ\_RULES | md.MdSysLookupBusobjRules | Complete | 1 | | MD\_SYS\_LOOKUP\_CALC\_FLAG | md.MdSysLookupCalcFlag | Complete | 0 | | MD\_SYS\_LOOKUP\_CARRYOVER\_CASE | md.MdSysLookupCarryoverCase | Complete | 0 | | MD\_SYS\_LOOKUP\_CARRYOVER\_OPER | md.MdSysLookupCarryoverOper | Complete | 0 | | MD\_SYS\_LOOKUP\_DATA\_TYPE | md.MdSysLookupDataType | Complete | 0 | | MD\_SYS\_LOOKUP\_DISPLAY\_TYPE | md.MdSysLookupDisplayType | Complete | 0 | | MD\_SYS\_LOOKUP\_INIT\_HINT | md.MdSysLookupInitHint | Complete | 0 | | MD\_SYS\_LOOKUP\_LINK\_TYPE | md.MdSysLookupLinkType | Complete | 0 | | MD\_SYS\_OW\_TEMPLATE\_DEFAULT | md.MdSysOwTemplateDefault | Complete | 0 | | MD\_SYS\_PARAM\_SETTINGS | md.MdSysParamSettings | Complete | 0 | | MD\_SYS\_PICKLIST\_COLS | md.MdSysPicklistCols | Complete | 0 | | MD\_SYS\_PICKLIST\_DISPLAY\_COLS | md.MdSysPicklistDisplayCols | Complete | 0 | | MD\_SYS\_REG\_MAPPING | md.MdSysRegMapping | Complete | 0 | | MD\_SYS\_REPLIST | md.MdSysReplist | Complete | 1 | | MD\_SYS\_REPLIST\_SECT\_CATALOG | md.MdSysReplistSectCatalog | Complete | 1 | | MD\_SYS\_REPORT\_TYPE | md.MdSysReportType | Complete | 0 | | MD\_SYS\_RUSHMORE\_LOOKUP | md.MdSysRushmoreLookup | Complete | 0 | | MD\_SYS\_SETTINGS | md.MdSysSettings | Complete | 0 | | MD\_SYS\_SPREADSHEETS | md.MdSysSpreadsheets | Complete | 0 | | MD\_SYS\_SPREADSHEET\_COLS | md.MdSysSpreadsheetCols | Complete | 0 | | MD\_SYS\_SYNC\_STATUS | md.MdSysSyncStatus | Complete | 0 | | MD\_SYS\_TABLE\_DEF | md.MdSysTableDef | Complete | 3 | | MD\_SYS\_TABLE\_RELATIONS | md.MdSysTableRelations | Complete | 1 | | MD\_TARGET\_COORD\_TYPE | md.MdTargetCoordType | Complete | 0 | | MD\_TARGET\_GEOMETRY | md.MdTargetGeometry | Complete | 0 | | MD\_TARGET\_POINT\_TYPE | md.MdTargetPointType | Complete | 0 | | MD\_TARGET\_TYPE | md.MdTargetType | Complete | 0 | | MD\_UTIL\_DEPTH\_DATUM | md.MdUtilDepthDatum | Complete | 0 | | MD\_UTIL\_DESCENT | md.MdUtilDescent | Complete | 0 | | MD\_UTIL\_FIELD\_ATTR | md.MdUtilFieldAttr | Complete | 1 | | MD\_VERTICAL\_COORD\_TYPE | md.MdVerticalCoordType | Complete | 0 | | MD\_VSA\_TYPE | md.MdVsaType | Complete | 0 | | MD\_VS\_ORIGIN | md.MdVsOrigin | Complete | 0 | | MD\_WELLBORE\_SECT | md.MdWellboreSect | Complete | 1 | | MD\_WELL\_COORD\_TYPE | md.MdWellCoordType | Complete | 0 | | MT\_SITE\_PROP\_OVERRIDES | mt.MtSitePropOverrides | Complete | 0 | | MT\_SYS\_MODULES | mt.MtSysModules | Complete | 1 | | MT\_SYS\_MODULE\_STATES | mt.MtSysModuleStates | Complete | 0 | | MT\_SYS\_MODULE\_WIZARD | mt.MtSysModuleWizard | Complete | 1 | | NT\_WELLCASE\_DET | nt.NtWellcaseDet | Complete | 0 | | PR\_VISUAL\_TRAJECTORY | pr.PrVisualTrajectory | Complete | 0 | | RT\_LOG\_CURVE\_BLOB | rt.RtLogCurveBlob | Complete | 0 | | RT\_LOG\_CURVE\_BLOBS | rt.RtLogCurveBlobs | Complete | 0 | | RT\_MEASURE\_LABEL\_MAP | rt.RtMeasureLabelMap | Complete | 0 | | RT\_SETTINGS | rt.RtSettings | Complete | 0 | | RT\_TRANSLATION | rt.RtTranslation | Complete | 0 | | TU\_ABANDONMENT\_TIME | tu.TuAbandonmentTime | Complete | 0 | | TU\_BIT\_COST | tu.TuBitCost | Complete | 0 | | TU\_BIT\_COST\_CATALOG | tu.TuBitCostCatalog | Complete | 0 | | TU\_CASE\_ALLOWABLE\_SIZE | tu.TuCaseAllowableSize | Complete | 0 | | TU\_CASE\_ANNULUS\_FLUID | tu.TuCaseAnnulusFluid | Complete | 0 | | TU\_CASE\_ASSEMBLY\_PARAMETER | tu.TuCaseAssemblyParameter | Complete | 0 | | TU\_CASE\_CORE | tu.TuCaseCore | Complete | 0 | | TU\_CASE\_DRILLING\_PARAMETER | tu.TuCaseDrillingParameter | Complete | 0 | | TU\_CASE\_PARAMETER | tu.TuCaseParameter | Complete | 0 | | TU\_CASE\_PARAMETER\_SEQ | tu.TuCaseParameterSeq | Complete | 0 | | TU\_CASE\_RESTRICTION | tu.TuCaseRestriction | Complete | 0 | | TU\_CASE\_ROP\_INTERVAL | tu.TuCaseRopInterval | Complete | 0 | | TU\_CASE\_USER\_PARAMETER | tu.TuCaseUserParameter | Complete | 0 | | TU\_COMPLETION\_COST | tu.TuCompletionCost | Complete | 0 | | TU\_COMPLETION\_TIME | tu.TuCompletionTime | Complete | 0 | | TU\_COMP\_TEMP\_DERATION\_POINT | tu.TuCompTempDerationPoint | Complete | 0 | | TU\_COST\_CATEGORY | tu.TuCostCategory | Complete | 0 | | TU\_CUSTOM\_LOAD\_PROFILE | tu.TuCustomLoadProfile | Complete | 0 | | TU\_DRILLING\_OPERATION | tu.TuDrillingOperation | Complete | 1 | | TU\_DRILLING\_OP\_PARAMETER | tu.TuDrillingOpParameter | Complete | 1 | | TU\_EPP\_PARAMETERS | tu.TuEppParameters | Complete | 1 | | TU\_EQUIPMENT\_COST | tu.TuEquipmentCost | Complete | 0 | | TU\_FLUID\_CATALOG | tu.TuFluidCatalog | Complete | 4 | | TU\_FLUID\_HC\_PROPERTY | tu.TuFluidHcProperty | Complete | 1 | | TU\_FLUID\_PRESSURE | tu.TuFluidPressure | Complete | 2 | | TU\_FLUID\_PRESS\_PROPERTY | tu.TuFluidPressProperty | Complete | 1 | | TU\_FLUID\_PVT\_PROPERTY | tu.TuFluidPvtProperty | Complete | 1 | | TU\_FLUID\_VLE\_HEAVY\_COMPONENT | tu.TuFluidVleHeavyComponent | Complete | 1 | | TU\_HANGER\_COST | tu.TuHangerCost | Complete | 0 | | TU\_HANGER\_COST\_CATALOG | tu.TuHangerCostCatalog | Complete | 0 | | TU\_LOAD\_HEADER | tu.TuLoadHeader | Complete | 2 | | TU\_LOAD\_PARAMETERS | tu.TuLoadParameters | Complete | 1 | | TU\_LOAD\_PROFILE | tu.TuLoadProfile | Complete | 0 | | TU\_LOG\_COST | tu.TuLogCost | Complete | 0 | | TU\_MAX\_ALLOWABLE\_WEAR | tu.TuMaxAllowableWear | Complete | 0 | | TU\_MISC\_COST | tu.TuMiscCost | Complete | 0 | | TU\_MISC\_TANGIBLE\_COST | tu.TuMiscTangibleCost | Complete | 0 | | TU\_MMS\_APD | tu.TuMmsApd | Complete | 1 | | TU\_MMS\_APD\_DETAIL | tu.TuMmsApdDetail | Complete | 1 | | TU\_OPERATION\_COST | tu.TuOperationCost | Complete | 0 | | TU\_SIZE\_RUN\_SPEED | tu.TuSizeRunSpeed | Complete | 0 | | TU\_SIZE\_RUN\_SPEED\_CATALOG | tu.TuSizeRunSpeedCatalog | Complete | 0 | | TU\_SPECIAL\_OPERATION | tu.TuSpecialOperation | Complete | 0 | | TU\_TEMP\_DERATION\_POINT | tu.TuTempDerationPoint | Complete | 1 | | TU\_TEMP\_DERATION\_SCHED | tu.TuTempDerationSched | Complete | 1 | | TU\_TOOL\_PASSAGE | tu.TuToolPassage | Complete | 0 | | TU\_TUBULAR\_COST | tu.TuTubularCost | Complete | 0 | | TU\_TUBULAR\_COST\_CATALOG | tu.TuTubularCostCatalog | Complete | 0 | | TU\_WEAR\_PREDICTION | tu.TuWearPrediction | Complete | 0 | | UMS\_SYS\_CLASS\_ASSIGNMENTS | um.UmsSysClassAssignments | Complete | 2 | | UMS\_SYS\_SETTINGS | um.UmsSysSettings | Complete | 0 | | UMS\_SYS\_UNIT\_CLASSES | um.UmsSysUnitClasses | Complete | 2 | | UMS\_SYS\_UNIT\_MEASURES | um.UmsSysUnitMeasures | Complete | 2 | | UMS\_SYS\_UNIT\_SYSTEMS | um.UmsSysUnitSystems | Complete | 3 | | UMS\_SYS\_UNIT\_SYSTEMS\_HEADER | um.UmsSysUnitSystemsHeader | Complete | 1 | | UMS\_SYS\_UNIT\_TYPES | um.UmsSysUnitTypes | Complete | 0 | | UMS\_TU\_CLASS\_MAPPING | um.UmsTuClassMapping | Complete | 0 | | UMS\_TU\_MAPPING | um.UmsTuMapping | Complete | 0 | | WC\_ACTIVITY\_CONFIG | wc.WcActivityConfig | Complete | 0 | | WC\_COST\_ITEM | wc.WcCostItem | Complete | 1 | | WC\_COST\_LIST | wc.WcCostList | Complete | 4 | | WC\_COST\_PHASE\_ASSOCIATION | wc.WcCostPhaseAssociation | Complete | 1 | | WC\_CURRENCY\_TYPE | wc.WcCurrencyType | Complete | 1 | | WC\_DAY\_COST | wc.WcDayCost | Complete | 1 | | WC\_DAY\_DEPTH | wc.WcDayDepth | Complete | 1 | | WC\_DESIGN\_SEQUENCE | wc.WcDesignSequence | Complete | 0 | | WC\_DETERMINISTIC\_ACTIVITY | wc.WcDeterministicActivity | Complete | 1 | | WC\_DETERMINISTIC\_EVENT | wc.WcDeterministicEvent | Complete | 7 | | WC\_DETERMINISTIC\_HOLE\_SECTION | wc.WcDeterministicHoleSection | Complete | 2 | | WC\_EVENT\_LIST | wc.WcEventList | Complete | 1 | | WC\_OFFSET\_ACTIVITY | wc.WcOffsetActivity | Complete | 1 | | WC\_OFFSET\_PHASE\_DURATION | wc.WcOffsetPhaseDuration | Complete | 1 | | WC\_OFFSET\_WELLBORE | wc.WcOffsetWellbore | Complete | 3 | | WC\_PHASE\_CONFIG | wc.WcPhaseConfig | Complete | 0 | | WC\_PHASE\_LIST | wc.WcPhaseList | Complete | 2 | | WC\_RISK\_LIST | wc.WcRiskList | Complete | 2 | | WC\_RISK\_PHASE\_ASSOCIATION | wc.WcRiskPhaseAssociation | Complete | 1 | | WP\_ADDITIVES | wp.WpAdditives | Complete | 0 | | WP\_BHA\_PARAMS | wp.WpBhaParams | Complete | 0 | | WP\_BHA\_WOB\_ROP | wp.WpBhaWobRop | Complete | 0 | | WP\_CASE\_ANNOTATIONS | wp.WpCaseAnnotations | Complete | 0 | | WP\_CASE\_BOOSTER\_PUMP | wp.WpCaseBoosterPump | Complete | 0 | | WP\_CASE\_CEM\_FOAM\_DATA\_ARR | wp.WpCaseCemFoamDataArr | Complete | 1 | | WP\_CASE\_CEM\_FOAM\_SCHD | wp.WpCaseCemFoamSchd | Complete | 0 | | WP\_CASE\_CEM\_FOAM\_SCHD\_ARR | wp.WpCaseCemFoamSchdArr | Complete | 0 | | WP\_CASE\_CEM\_FOAM\_SCHD\_ARR2 | wp.WpCaseCemFoamSchdArr2 | Complete | 0 | | WP\_CASE\_CEM\_JOB\_DATA | wp.WpCaseCemJobData | Complete | 0 | | WP\_CASE\_CEM\_JOB\_DATA\_ARR | wp.WpCaseCemJobDataArr | Complete | 1 | | WP\_CASE\_CEM\_JOB\_DATA\_PRES | wp.WpCaseCemJobDataPres | Complete | 0 | | WP\_CASE\_CENTRALIZER\_SCHD | wp.WpCaseCentralizerSchd | Complete | 0 | | WP\_CASE\_CENTRALIZER\_SCHD\_INT | wp.WpCaseCentralizerSchdInt | Complete | 0 | | WP\_CASE\_CIRC\_SYSTEM | wp.WpCaseCircSystem | Complete | 0 | | WP\_CASE\_CSA\_BOUNDARY\_COND | wp.WpCaseCsaBoundaryCond | Complete | 0 | | WP\_CASE\_CSA\_BOUNDARY\_TYPES | wp.WpCaseCsaBoundaryTypes | Complete | 0 | | WP\_CASE\_CSA\_PARAMS | wp.WpCaseCsaParams | Complete | 0 | | WP\_CASE\_ECCENTRICITY | wp.WpCaseEccentricity | Complete | 0 | | WP\_CASE\_PACKER\_ANALYSIS | wp.WpCasePackerAnalysis | Complete | 0 | | WP\_CASE\_PUMP | wp.WpCasePump | Complete | 0 | | WP\_CASE\_PUMP\_SLOW | wp.WpCasePumpSlow | Complete | 0 | | WP\_CASE\_RANGE | wp.WpCaseRange | Complete | 0 | | WP\_CASE\_RT\_HYD | wp.WpCaseRtHyd | Complete | 0 | | WP\_CASE\_RT\_TDA | wp.WpCaseRtTda | Complete | 0 | | WP\_CASE\_SOIL\_PROP | wp.WpCaseSoilProp | Complete | 0 | | WP\_CASE\_SURFACE | wp.WpCaseSurface | Complete | 0 | | WP\_CASE\_TORT\_INT | wp.WpCaseTortInt | Complete | 0 | | WP\_CASE\_WBSIMULATOR\_ANAL | wp.WpCaseWbsimulatorAnal | Complete | 0 | | WP\_CUSTOM\_BASE\_FLUID | wp.WpCustomBaseFluid | Complete | 0 | | WP\_DRUM\_DIAMETER | wp.WpDrumDiameter | Complete | 1 | | WP\_DYNAMICPROPERTY | wp.WpDynamicproperty | Complete | 1 | | WP\_DYNAMICPROPERTY\_HEADER | wp.WpDynamicpropertyHeader | Complete | 1 | | WP\_FLUID\_GAS | wp.WpFluidGas | Complete | 1 | | WP\_FLUID\_GAS\_PROPERTIES | wp.WpFluidGasProperties | Complete | 1 | | WP\_FLUID\_TEMP | wp.WpFluidTemp | Complete | 1 | | WP\_FLUID\_TEMP\_FANN\_DATA | wp.WpFluidTempFannData | Complete | 1 | | WP\_FLUID\_TUNED\_RECIPE | wp.WpFluidTunedRecipe | Complete | 0 | | WP\_FLUID\_TUNED\_SPACER | wp.WpFluidTunedSpacer | Complete | 0 | | WP\_GAS | wp.WpGas | Complete | 0 | | WP\_HYD\_ACTVAL | wp.WpHydActval | Complete | 0 | | WP\_HYD\_OPERATIONAL | wp.WpHydOperational | Complete | 0 | | WP\_HYD\_OPTIONS | wp.WpHydOptions | Complete | 0 | | WP\_HYD\_PARAMS | wp.WpHydParams | Complete | 0 | | WP\_KILL\_SHEET\_GEN | wp.WpKillSheetGen | Complete | 0 | | WP\_MAST\_HEIGHT | wp.WpMastHeight | Complete | 1 | | WP\_RPT\_OPTIONS | wp.WpRptOptions | Complete | 0 | | WP\_SRG\_PARAMS | wp.WpSrgParams | Complete | 0 | | WP\_SRG\_RECIPROCATION | wp.WpSrgReciprocation | Complete | 0 | | WP\_SRG\_SURGE | wp.WpSrgSurge | Complete | 0 | | WP\_SRG\_SWAB\_SURGE | wp.WpSrgSwabSurge | Complete | 0 | | WP\_STATUS\_MESSAGE | wp.WpStatusMessage | Complete | 0 | | WP\_STUCKPIPE\_PARAMS | wp.WpStuckpipeParams | Complete | 0 | | WP\_SURVEY\_IMPORT | wp.WpSurveyImport | Complete | 0 | | WP\_TDA\_ACTLOAD | wp.WpTdaActload | Complete | 0 | | WP\_TDA\_ANNULUS\_FLUID\_GRAD | wp.WpTdaAnnulusFluidGrad | Complete | 0 | | WP\_TDA\_DISCRETE\_PARAMS | wp.WpTdaDiscreteParams | Complete | 0 | | WP\_TDA\_DRAGCHART | wp.WpTdaDragchart | Complete | 0 | | WP\_TDA\_DRAGCHART\_COF | wp.WpTdaDragchartCof | Complete | 0 | | WP\_TDA\_DRAGCHART\_MAN\_COF | wp.WpTdaDragchartManCof | Complete | 0 | | WP\_TDA\_DRAGCHART\_SENS | wp.WpTdaDragchartSens | Complete | 0 | | WP\_TDA\_FRD | wp.WpTdaFrd | Complete | 1 | | WP\_TDA\_FRD\_ALL | wp.WpTdaFrdAll | Complete | 1 | | WP\_TDA\_FRD\_ALL\_FORCE | wp.WpTdaFrdAllForce | Complete | 1 | | WP\_TDA\_FRD\_FORCE | wp.WpTdaFrdForce | Complete | 1 | | WP\_TDA\_MANUAL\_COF | wp.WpTdaManualCof | Complete | 0 | | WP\_TDA\_OPTIONS | wp.WpTdaOptions | Complete | 0 | | WP\_TDA\_PARAMS | wp.WpTdaParams | Complete | 0 | | WP\_TDA\_RSRLSS\_DRAGCHART | wp.WpTdaRsrlssDragchart | Complete | 0 | | WP\_TDA\_RSRLSS\_FRD | wp.WpTdaRsrlssFrd | Complete | 1 | | WP\_TDA\_RSRLSS\_FRD\_FORCE | wp.WpTdaRsrlssFrdForce | Complete | 1 | | WP\_TDA\_RSRLSS\_INN\_ANN\_FLUID | wp.WpTdaRsrlssInnAnnFluid | Complete | 0 | | WP\_TDA\_RSRLSS\_OUT\_ANN\_FLUID | wp.WpTdaRsrlssOutAnnFluid | Complete | 0 | | WP\_TDA\_STRING\_FLUID\_GRAD | wp.WpTdaStringFluidGrad | Complete | 0 | | WP\_TDA\_STR\_RSRLSS\_FLUID\_GRAD | wp.WpTdaStrRsrlssFluidGrad | Complete | 0 | | WP\_UBD\_PARAMS | wp.WpUbdParams | Complete | 0 | | WP\_UBD\_PARASITE\_STRING | wp.WpUbdParasiteString | Complete | 0 | | WP\_WCN\_PARAMS | wp.WpWcnParams | Complete | 0 | | WP\_WORKSPACE | wp.WpWorkspace | Complete | 1 | | WP\_WPC | wp.WpWpc | Complete | 1 | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/edm/edm_server_requirements Hardware, software, and network requirements for running the Cognite EDM extractor on physical servers, virtual machines, or cloud environments. ## Memory * **Minimum**: 4GB RAM / 30 GB storage * **Recommended**: 8GB RAM / 30 GB storage (for intensive extractions or large amounts of historical records) ## CPU * **Minimum**: 2-core CPU * **Recommended**: 4-core CPU (for intensive extractions or large amounts of historical records) ## Software * **For .jar installation**: Java Runtime Environment is required * **For Docker container**: Docker environment is required ## Network access * Username and password for connecting to the Halliburton DSIS/EDM environment * Network access with firewall openings to the CDF cluster and CDF project # Set up the EDM extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/edm/edm_setup Install and configure the Cognite EDM extractor as a Java .jar file or Docker container to extract data from Landmark EDM to CDF. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/edm/edm_server_requirements) for the extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#edm-extractor) for the extractor to write data to the respective CDF destination resources. * Create a `config` directory and add a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/edm/edm_configuration). The file must be in YAML format. * If you're running the extractor as a Docker image, you need access to [Cognite's internal Docker registry](http://eu.gcr.io/cognite-registry/edm-connector). Contact [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new) to get access. ## Run as a .jar file Navigate to **Data fusion** > **Integrate** > **Extractors** to download the .jar file. Run the .jar file with the following command: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} java -jar path/to/cognite-edm-extractor.jar path/to/config.yml ``` ## Run as a Docker container Download the Docker image from [Cognite's internal Docker registry](http://eu.gcr.io/cognite-registry/edm-connector): ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} gcloud auth configure-docker gcloud auth login --no-launch-browser ``` Enter this `docker run` statement: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} docker run -v path_to_your_config_file/config.yml:/config/config.yml \ -it eu.gcr.io/cognite-registry/edm-connector:latest ``` * `docker run` starts a new container. * `-v /path/to/config.yml:/config/config.yml` shares the file at `/path/to/config.yml` with the container, putting it at `/config/config.yml`, which is the default configuration file location for the Docker image. * `eu.gcr.io/cognite-registry/edm-connector:` is the name of the image. Replace `` with the version you'll run. Cognite strongly recommends running the latest version. ## Load data incrementally You can set up the extractor for full-load extractions or incremental extractions. You load data incrementally by adding the [delta-supported EDM entities](/cdf/integration/guides/extraction/edm/edm_entities) to the `entities` configuration parameter and set a schedule for checking incremental EDM entities with the `gracePeriod` configuration parameter. The extractor creates a CDF RAW table for each EDM entity. A full extraction is triggered according to the configured schedule in the `crontabComplete` configuration parameter for listed entities that don't support incremental extraction. EDM doesn't expose a create or update time for each row. ## Navigating the explored data The extractor sends the EDM records to CDF RAW, creating a table for each EDM entity defined in the extractor configuration. In addition, it created two tables to hold the state of the extractor: * `CdfEdmObjectInfos`: Stores a list of the unique objects across all entities that have been extracted from EDM. When an object changes, a new revision is added to the entity-specific table, and the location link column for that entry is updated to point to the new version. This table also holds the state of whether the object has been deleted in the source EDM installation. This is periodically checked if you've configured the `consistencySchedule` configuration parameter. * `CdfEdmExtractionStates`: Stores the information about the extraction and when it last successfully ran. There are two methods to process the extracted records: Iterate through the rows in `CdfEdmObjectInfos` and filter on the entity types relevant for the client (the entity type is in the `type` field). Then for each of the rows, retrieve the serialized object by looking up the key found in `locationLink` in the matching entity RAW table. This ensures that the last version is retrieved. The `state` field specifies whether the entity instance is `ACTIVE` or `DELETED`. Iterate through the rows of the entity-specific table and process them directly. This returns all versions of the entity instances the extractor has seen. The client should verify whether an object is deleted by looking up the matching entry in `CdfEdmObjectInfos`. To find the key to lookup in `CdfEdmObjectInfos`, remove the trailing version from the key in the entity-specific version. This is an underscore followed by a positive number. Once a full iteration has been done, the client should save the maximum value for `lastUpdatedTime` encountered for rows, and then on the next iteration query for new rows since this time by setting the `minLastUpdatedTime` query parameter to the value minus 1 (since `minLastUpdatedTime` is exclusive). For entities that support incremental processing, the value persisted for the entity in `CdfEdmExtractionStates` is used to query for new instances on each iteration. To allow for clock skew between the EDM installation and the client, a buffer period is subtracted from the timestamp before it's used to query for objects that have a newer update or create value in EDM. Deleting the row entry for a specific EDM entity from the `CdfEdmExtractionStates`, and restarting the extractor, will make the extractor re-process an incremental EDM entity from scratch. ### Using the extracted data The records are persisted as [protobuf](https://developers.google.com/protocol-buffers), in the `edmObject` column for each row in the entity-specific table. Protobuf retains both the typing of the properties and the precision of the values. This is an example of what a `CD_WELLBORE` extracted record looks like in CDF RAW: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "edmObject": "1nrwSJrvVfVyqSGRkPG1WSK0yRVwbvpmSMFH5inySCnFW9btRSQDNNPZYPNDHANNNVjfbONjbOoqbONjbOorbOPDbUD09AHRSGH4VPPDbUD09AHRSGH5VPODbQMTIaftVQPtSghtVSQDNNPZYXNtZXNJ3nNtZXNIavNtZXNJ2nNk8XUKOyqT1wMFuxLKcfoJguLaOjpzDjASkjMKEgL2HcdtZXMRSjIT1nrwSJriVQPNbToJI0MKWmttDSPtAxMJrnONLVd8sUzNndONZXNJ2lONZXNJ3XONZXNJ3FONZXNJ3vONcmZIyWGz9dHH9c8tDQPtSgvtHQPtSgltHQPtSg0tHQPtSg2tHTPZsh85pT4tHQPtSgvtLQPtSgjtLQPtSg0tLSPtAxMJsvOtHANNNVjhbTUjbqp2yxn3IhXTEurzkgn2SvpUOlMQN0KUAcMTg1ovalOtZXNHi6OtZXNH6vOjZXNJ2dOmLXARIRAGNtXvOSHSAUYH5ipvOGAwVtZwNjZFNiVSIHGFO" } ``` To explore the extracted records, you can use the [cognite-edm-data](https://pypi.org/project/cognite-edm-data) Python library to decode the protobuf objects into Python objects. For other languages (such as Java and C#), the extractor distribution includes the protobuf specification used to persist the objects in the file `distribution-protobuf.proto`. You can use this specification as input for the [protoc](https://developers.google.com/protocol-buffers/docs/proto#generating) tool to generate code for decoding the persisted objects. Below you will find an example Python code snippet for how to decode EDM protobuf objects. This example converts the serialized object into a Python dictionary object. ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import cognite.edm.connector.native.edm_native_v1_pb2 as edm_protobuf from base64 import b64decode from google.protobuf.json_format import MessageToDict def decode_object(data): decoded = b64decode(data) edm = edm_protobuf.EDMObjects() message = edm.FromString(decoded) return MessageToDict(message.objects[0], preserving_proto_field_name=True) # function expects CDF client, raw database name and edm raw table name as parameters def read_objects(client, db_name, table_name): raw = [ row.columns for row in client.raw.rows( db_name=db_name, table_name=table_name, limit=None, ) ] return [ decode_object(row['edmObject']) for row in raw ] ``` # Event Hub extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/eventhub Learn how to use the Cognite Event Hub extractor to connect to Azure Event Hub and extract data using the AMQP protocol.Cognite hosts the Event Hub extractor, so you don't have to download or install anything. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#cognite-event-hub-extractor) to *create* an Event Hub extractor and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models in the target CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). ## Deploy the extractor Navigate to Data fusion > Integrate > Extractors. Locate the **Cognite Event Hub extractor** and select **Set up extractor**. Get a connection string that you'll need to set up the Event Hub extractor: 1. Locate your Event Hub in Azure. 2. Navigate to Hub Settings > Built-in endpoints > Event Hub compatible endpoint. Typically, it's sufficient to create a single **job** per Event Hub **source**. An Event Hub is equivalent to a single **topic** on other streaming platforms. You can't filter data received from an Event Hub. When you connect to an Event Hub, you specify a **ConsumerGroup**. We recommend to create a new consumer group for each application that reads from your Event Hub. ## Message formats Azure Event Hub is a data streaming service on Azure that supports payloads in any format. The Cognite Event Hub extractor supports several [predefined message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings). If you want to define your own custom mapping of Event Hub messages, see [custom data formats for hosted extractors](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). Custom formats used with Event Hub jobs will receive an `input` argument containing the message in a JSON format. # About the Cognite File extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/file Learn about the Cognite File extractor and how to connect it to local file systems, cloud storage, and network sharing protocols to upload files into Cognite Data Fusion (CDF). **Deploy with the Cognite Toolkit:** The [Foundation deployment pack](/cdf/deploy/cdf_toolkit/references/packages/foundation) includes a `cdf_files_extractor` module with ready-made extraction pipeline configs — use it instead of configuring the extractor pipeline by hand. The Cognite File extractor connects to online file storage services, such as SharePoint Online Document libraries, Google cloud storage, and Azure blob storage, and network sharing protocols, such as FTP, FTPS, and SFTP. The extractor uploads files into the CDF files service. You can [configure the extractor](/cdf/integration/guides/extraction/configuration/file-extractor), for instance with the file types you want to extract and set filters for maximum file size limit. The extractor can also upload files to a local file system. The extractor is available as a [Windows and Linux executable file](/cdf/integration/guides/extraction/file/file_setup). # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/file/file_requirements Hardware, software, and network requirements for running the Cognite File extractor. You can run the extractor on a physical server, on a virtual machine on-premises, or in the cloud. These requirements depend on the number and size of files. ## Memory The server needs a minimum of 1 GB RAM and an extra 32 GB HD for logging and storage. ## CPU The server needs a minimum of 1-core CPU. ## Software An operating system that runs on Windows or Linux. ## Network access Network access with firewall openings to the CDF cluster and a CDF project. ## CDF access You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). # Set up the Cognite File extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/file/file_setup Install and configure the Cognite File extractor on Windows and Linux systems. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/file/file_requirements) for the extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#file-extractor) in a CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). * Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime. * To create and monitor the extractor as an **integration** (public preview), see [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration). You can also navigate to Data fusion > Integrate > Extractors to download the Windows executable file. Set Modify permission under Properties in the installation folder to avoid permission issues. * Create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/configuration/file-extractor). The file must be in YAML format. ## Run as a Windows standalone executable file Download the `file_extractor-{VERSION}-win32.exe` file via the download links available from the Cognite File extractor section on the **Extract data** page in the CDF user interface. Save the file in a folder. Open a command line window and run the file with a configuration file as an argument. In this example, the configuration file is named `config.yml` and saved in the same folder as the executable file: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .\file-extractor-standalone-{VERSION}-win32.exe ./config.yml ``` ## Install and run the extractor ### Install using Windows installer Download the `file-extractor-{VERSION}-win32.msi` file via the download links available from the Cognite File extractor section on the **Extract data** page in the CDF user interface. Save the file in a folder. Navigate to the folder where you saved the file. Double-click the file, or type the below command in a command line window to install the extractor. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .\file-extractor-{VERSION}.msi ``` ### Run as standalone executable Download the `file_extractor-{VERSION}-win32.exe` file via the download links available from the Cognite File extractor section on the **Extract data** page in the CDF user interface. Save the file in a folder. Open a command line window and run the file with a configuration file as an argument. In this example, the configuration file is named `config.yml` and saved in the same folder as the executable file: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .\file-extractor-standalone-{VERSION}-win32.exe ./config.yml ``` ### Run as Windows service Download the `file-extractor-service-{VERSION}-win32.exe` file via the download links available from the Cognite File extractor section on the **Extract data** page in the CDF user interface. Save the file to the same directory as a configuration file named `config.yaml`. This must be the exact name of the file. As an administrator, open up the command line window in the folder you placed the executable file and the configuration file and run the following command: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .\file-extractor-winservice-{VERSION}-win32.exe install ``` Open the **Services** app in Windows and find the **Cognite File Extractor Executor** service. Right-click the service and select **Properties**. Configure the service per your needs/requirements. ### Run as standalone executable Download the `file_extractor-standalone-{VERSION}-linux` file via the download links available from the Cognite File extractor section on the **Extract data** page in the CDF user interface. Save the file in a directory. Open a terminal window, and run the file with its configuration file as an argument. In this example, the configuration file is named `config.yml` and saved in the same folder as the executable file: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ./file-extractor-standalone-{VERSION}-linux ./config.yml ``` # Connect to SharePoint Online Source: https://docs.cognite.com/cdf/integration/guides/extraction/file/file_sharepoint_setup Configure the Cognite File extractor to connect to SharePoint Online sources. The configuration is subject to change depending on permissions, restricted access on the SharePoint level, etc. To register SharePoint Online through **Integrations** (public preview), create a File on-premises integration. See [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration). ## Register an application [Follow the steps to register an application](/cdf/access/entra/guides/register_custom_webapp) that the Cognite File extractor will use to connect to SharePoint Online. [Set up and create groups](/cdf/integration/guides/extraction/admin_oidc#step-2-create-a-group-in-microsoft-entra-id-and-add-the-registered-app-as-its-member). Assign relevant API permissions to use Microsoft Graph API. The example has the minimal setup for the File extractor to read all SharePoint Online sites. ``` Sites.Read.All (Type: Application) User.Read (Type: Delegated) ``` App Registration API permissions showing Sites.Read.All and User.Read permissions ## SharePoint app permissions After creating the App Registration, add its details to the SharePoint administration app registration. To create a new SharePoint app permission, go to this URL [https://YOUR-SHAREPOINT-NAME-admin.sharepoint.com/\_layouts/15/AppInv.aspx](https://YOUR-SHAREPOINT-NAME-admin.sharepoint.com/_layouts/15/AppInv.aspx), where `YOUR-SHAREPOINT-NAME` should be changed. Make sure to use your Microsoft 365 admin account. Enter your Application (client) ID in the App Id field and select Lookup. You should see the App registration details filled out. SharePoint App permission configuration screen with App Id lookup field Enter App Domain and Redirect URL of your App registration. For a local execution of the File extractor, enter the values of a localhost domain: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} App Domain: www.localhost.com Redirect URL: https://www.localhost.com/default.aspx ``` Add the App's Permission Request XML to configure the SharePoint permission level. This may vary from every configuration scenario. The example gives the minimal XML with the `Read` permissions to all SharePoint sites. ```xml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ``` The configuration is subject to change depending on permissions, restrict access on the SharePoint level, etc. Select Create and then confirm that you trust SharePoint. ## Run extractor Once you register the application and add the SharePoint app permissions, you can extract files from SharePoint Online. Go to **Data fusion** > **Integrate** > **Extractors** > **Cognite File extractor** > **Download extractor**. In the downloaded folder, find and edit the `example-sharepoint.yaml` configuration template by setting the App registration information and the related SharePoint configuration parameters. Run the extractor. The example has a successful execution log. ``` 2023-11-21 09:06:16.564 UTC [INFO ] ThreadPoolExecutor-0_0 - All files processed. 5 files uploaded 2023-11-21 09:06:16.564 UTC [INFO ] ThreadPoolExecutor-0_0 - Job "FileExtractor (trigger: interval[0:00:10], next run at: 2023-11-21 10:06:20 CET)" run successfully 2023-11-21 09:06:20.207 UTC [INFO ] ThreadPoolExecutor-0_0 - Running job "FileExtractor (trigger: interval[0:00:10], next run at: 2023-11-21 10:06:30 CET)" (scheduled at 2023-11-21 10:06:20.184848+01:00) ``` Check the extracted files in **Data fusion** > **Data explorer**. ## Find the right paths in SharePoint Online You can configure the File extractor to extract data from various paths in SharePoint Online. It's possible to provide paths to sites, document libraries, folders, and individual files. ### Choose a URL or structured path In your extractor configuration, each SharePoint source is a path under `sharepoint_online.paths`. You can specify the path as a URL or as a structured path. | SharePoint structure | Path type | File extractor version | | - | - | - | | Root site, document library, folder, or file | URL | Any supported version | | One subsite under the root site | URL | Any supported version | | Two or more nested subsites | Structured | 2.12.0 or later | Use a URL when you can copy the location directly from the SharePoint address bar. Use a structured path when the site hierarchy has two or more nested subsites (for example, `site/subsite1/subsite2/library`). For structured path parameters, see [`sharepoint_structured_path`](/cdf/integration/guides/extraction/configuration/file-extractor#files.file-provider.sharepoint_online.paths.sharepoint_structured_path) in the configuration reference. ### Paths to sites You can get paths to sites from the address bar in your browser. Go to your SharePoint Online site, copy the URL from the address bar, and paste it into your configuration as a [`url`](/cdf/integration/guides/extraction/configuration/file-extractor#files.file-provider.sharepoint_online.paths.sharepoint_url_path) path. For example, `https://cogsp.sharepoint.com/sites/MySite`. URLs with `/teams/` instead of `/sites/` are valid for URL paths. Structured paths support `/sites/` only. ### Paths to nested subsites If your files live under two or more nested subsites, use a structured path in your configuration. URL paths do not support this hierarchy. For example, if the address bar shows `https://cogsp.sharepoint.com/sites/MySite/Subsite1/Subsite2/Shared%20Documents`, configure a structured path instead of a URL path: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} paths: - type: structured host: cogsp.sharepoint.com site: MySite sub_sites: Subsite1/Subsite2 library: Shared Documents ``` Separate each nested subsite in `sub_sites` with `/`. There is no depth limit. This feature requires Cognite File extractor **2.12.0** or later. ### Paths to document libraries Sites in SharePoint Online can contain one or more document libraries where you store your files. New sites generally have only a document library called `Documents`, but you can create any number of document libraries. To extract a single document library from a site, you need to find this document library URL. For a document library under a single subsite, you can use a URL path. Copy the URL from the address bar until `/Forms/AllItems.aspx`, for example, `https://cogsp.sharepoint.com/sites/MySite/Subsite1/Shared%20Documents`. For a library under two or more nested subsites, use a structured path. See [Paths to nested subsites](#paths-to-nested-subsites). Go to the document library you want to extract. Copy the beginning of the URL until `/Forms/AllItems.aspx`, for example, `https://cogsp.sharepoint.com/sites/MySite/Shared%20Documents`. ### Paths to files and folders To find paths to files and folders: In SharePoint Online, go to **Documents** and select the file or the folder you want to extract. Select **More actions** (...) > **Details**. Scroll down to **Paths** and select **Copy direct link** to copy the URL to your clipboard, for example, `https://cogsp.sharepoint.com/sites/MySite/Shared%20Documents/my%20folder/test.doc`. The copied URL matches the document library URL with the file or folder path added. The `recursive` setting traverses folders inside the resolved document library. It does not traverse into additional SharePoint subsites. ## Troubleshooting If extraction fails when you configure SharePoint paths, try the solutions below. The extractor returns this error when it cannot resolve the document library for a SharePoint path configured with `url:`. You may see it when: * The document library is missing from the URL or points to the wrong library. * The URL path includes **two or more nested subsites** before the document library. Cognite File extractor **2.12.0** and later uses the same error message for these URL path cases as earlier versions. Upgrade alone does not fix the error. You need a structured path for multilevel subsites. To fix the error: 1. Use Cognite File extractor **2.12.0** or later (required for structured paths). 2. Replace the URL path with a structured path that sets `host`, `site`, and `sub_sites`. See [Paths to nested subsites](#paths-to-nested-subsites). 3. For parameter details, see [`sharepoint_structured_path`](/cdf/integration/guides/extraction/configuration/file-extractor#files.file-provider.sharepoint_online.paths.sharepoint_structured_path) in the configuration reference. # Hosted extractors overview Source: https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors Learn about hosted extractors in Cognite Data Fusion (CDF) for live low latency data streams, including sources, jobs, destinations, mappings, and API-based email alerts when job status changes. A hosted extractor **job** reads from a **source**, transforms the data using a built-in format or a **mapping**, and writes to a **destination**. In order to create a hosted extractor, you need the `hostedextractors:READ` and `hostedextractors:WRITE` capabilities. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). ## Sources A hosted extractor **source** represents an external source system on the internet. The **source** resource in CDF contains all the information the extractor needs to connect to the external source system. A source can have many jobs, each streaming different data from the source system. ## Jobs A hosted extractor **job** represents the running extractor. Jobs produce logs and metrics that give the state of the job. A job can have nine different states: | State | Description | | - | - | | **Paused** | The job is temporarily stopped. | | **Waiting to start** | The job is temporarily stopped and pending start. This state typically only lasts a few seconds. | | **Stopping** | The job is running, but is supposed to stop. This state should only last for a few seconds at most. | | **Startup error** | The job failed to start and will not attempt to restart. Check the configuration settings for the job to resolve this state. | | **Connection error** | The job failed to connect to the source system and is currently retrying. | | **Connected** | The job is connected to the source system, but has not yet received any data. | | **Transform error** | The job is connected to the source system, received data but failed to transform and ingest the data into a CDF resource type. | | **Destination error** | The job successfully transformed data, but failed to ingest data into a CDF resource type. | | **Running** | The job is streaming data into CDF. | ### Job metrics Jobs report metrics on their current execution. The following metrics are currently reported: | Metric | Description | | - | - | | **Source messages** | The number of input messages received from the source system. | | **Transform failures** | The number of input messages that failed to transform. | | **Destination input values** | The number of messages that successfully transformed and were given to destinations for uploading to CDF. | | **Destination requests** | The number of requests made to CDF for this job. | | **Destination write failures** | The number of requests to CDF that failed for this job. | | **Destination skipped values** | The number of values that were invalid and were skipped before ingestion into CDF. | | **Destination failed values** | The number of values that were not written to CDF due to failed requests. | | **Destination uploaded values** | The number of values that were successfully ingested into CDF. | For the **Cognite MQTT extractor**, see [Message delivery and reliability](/cdf/integration/guides/extraction/mqtt#message-delivery-and-reliability) for details about message delivery during disconnections. ## Email notifications The feature described in this section is currently in [public preview](/cdf/product_feature_status) testing and is subject to change. Validate behavior in non-production environments first. Hosted extractor [jobs](/cdf/integration/guides/extraction/hosted_extractors#jobs) expose **state** and **metrics** in CDF. Email notifications tell operators when a job enters an operational state you care about (for example an error or paused state) or returns to a healthy state. Notifications are delivered through the **[Signals API](https://api-docs.cognite.com/20230101-beta/tag/Signals)**. Receive **email notifications** when a hosted extractor **job** changes status by creating **signal sinks** (email delivery targets) and **signal subscriptions** that match job status signals. Users with the appropriate [access capabilities](/cdf/access/guides/capabilities#job-email-notifications) can configure notifications for any user. Scope **All** lets you configure notifications broadly and **Current user** limits configuration to yourself. The Signals API is documented as **preview**. Send the `cdf-version` header your environment expects and review [Beta API reference](/api-reference/concepts/20230101-beta/beta-api) before you depend on this in production. ## Destinations A hosted extractor writes to a **destination**. The destination only contains credentials for CDF. Multiple jobs can share a single destination, in which case they will make requests together, reducing the number of requests made to the Cognite APIs. Metrics will still be reported individually. For permanent deployments, the **destination** should be given a dedicated set of credentials. The credentials given to a destination should only grant access to the CDF resources the extractor needs to write based on the job's mapping. See [Data formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings) for details on mappings. ### Destination sessions and capabilities When you create a destination, CDF stores a snapshot of the capabilities on the user or client credentials. Later changes to those capabilities do not apply to that destination. Create a new destination after you change those capabilities. Then point existing jobs at the new destination. You cannot delete the old destination while a job still uses it. ### Supported output types Depending on the job **format** or **mapping**, a job can write to these CDF resource types: * Datapoints and time series * Events * RAW table rows * Data model instances (nodes and edges) * Records in streams Writing to Records is a [public preview](/cdf/product_feature_status) feature. Use a [custom mapping](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings#records) with the `immutable_record` output type and set `beta` to `true`. For an end-to-end setup example, see [Build a stream-to-stream transformation pipeline](/cdf/dm/records/guides/records_getting_started#advanced-build-a-stream-to-stream-transformation-pipeline). This example covers source, destination, mapping, and job configuration for a hosted extractor that reads from one stream and writes transformed records to another. ### Access capabilities The CDF user creating an instance of a hosted extractor needs both the `hostedextractors:READ` and `hostedextractors:WRITE` capabilities. | Output type | Required capabilities | | - | - | | **Datapoints** | **timeseries:READ** and **timeseries:WRITE**. **datamodelinstances:WRITE** if you write to a data model extended from the `CogniteTimeSeries` data model type. | | **Timeseries** | **timeseries:READ** and **timeseries:WRITE**. **datamodelinstances:WRITE** if you write to a data model extended from the `CogniteTimeSeries` data model type. | | **Events** | **events:READ** and **events:WRITE**. **assets:READ** if you produce events linked to assets. | | **Raw** | **raw:WRITE** | | **Data Models** | **datamodelinstances:WRITE** | | **Records** | **streamrecords:WRITE** and **datamodels:READ** scoped to each space that contains a source container. | For extra security, you can scope the access of the extractor to the data sets/spaces it writes to. Granting **hostedextractors:WRITE** to a user may cause a privilege escalation. Any user that has access to create hosted extractors also has access to all the resources any configured **destinations** have access to. ## Mapping A mapping is a custom transformation, translating the source format to a format that can be ingested into CDF. Read more in [Custom data formats for hosted extractors](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). # Functions Source: https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/built_in_functions Complete reference of all built-in functions in Kuiper, the hosted extractor mapping language, for data transformation and manipulation. Use this reference when you write hosted extractor mappings in Kuiper. Each section documents one function's signature, behavior, and examples. For an overview of variables, operators, and lambda expressions, see [Mapping concepts](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts#lambda-expressions). ## acos `acos(x)` Return the inverse cosine of `x` in radians between 0 and pi. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} acos(0) -> 1.5707963267948966 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} acos(1) -> 0.0 ``` ## all `all(x)` Return `true` if all items in the array `x` are true. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [true, false, false, true].all() -> false ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [true, true, true, true].all() -> true ``` ## any `any(x)` Return `true` if any items in the array `x` are true. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [true, false, false, true].any() -> true ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [false, false, false, false].any() -> false ``` ## asin `asin(x)` Return the inverse sine of `x` in radians between -pi/2 and pi/2. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} asin(0) -> 0.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} asin(1) -> 1.5707963267948966 ``` ## atan `atan(x)` Return the inverse tangent of `x` in radians between -pi/2 and pi/2. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} atan(0) -> 0.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} atan(1) -> 0.7853981633974483 ``` ## atan2 `atan2(x, y)` Return the inverse tangent of `x`/`y` in radians between -pi and pi. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} atan2(3, 2) -> 0.982793723247329 ``` ## case `case(x, c1, r1, c2, r2, ..., (default))` Compare `x` to each of `c1`, `c2`, etc. and return the matching `r1`, `r2` of the first match. If no entry matches, a final optional expression can be returned as default. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} case("b", "a", 1, "b", 2, "c", 3, 0) -> 2 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} case("d", "a", 1, "b", 2, "c", 3, 0) -> 0 ``` ## ceil `ceil(x)` Return `x` rounded up to the nearest integer. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ceil(16.2) -> 17 ``` ## chars `chars(x)` Create an array of characters from a string. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "test".chars() -> ["t", "e", "s", "t"] ``` ## chunk `chunk(x, s)` Convert the list `x` into several lists of length at most `s` **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} chunk([1, 2, 3, 4, 5, 6, 7], 3) -> [[1, 2, 3], [4, 5, 6], [7]] ``` ## coalesce `coalesce(a, b, ...)` Return the first non-null value in the list of values. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} coalesce(null, "a", "b") -> "a" ``` ## concat `concat(x, y, ...)` Concatenate any number of strings. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} concat("Hello, ", "world!") -> "Hello, world!" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": concat("some-prefix:", input.tag) } ``` ## contains `contains(x, a)` Return `true` if the array or string `x` contains item `a`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].contains(4) -> true ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".contains("llo wo") -> true ``` ## cos `cos(x)` Return the cosine of `x`, where `x` is in radians. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cos(0) -> 1.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cos(3.141592653589793 / 2) -> 0.0 ``` ## digest `digest(a, b, ...)` Compute the SHA256 hash of the list of values. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} digest("foo", "bar", 123, [1, 2, 3]) -> lDN5G9Qz3fKZM6joQq+1OdF8P1rs2WYrgawlFXflqss= ``` ## distinct\_by `distinct_by(x, (a(, b)) => ...)` Return a list or object where the elements are distinct by the returned value of the given lambda function. The lambda function either takes list values, or object (value, key) pairs. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4, 5].distinct_by(x => x % 2) -> [1, 2] ``` ## ends\_with `ends_with(item, substring)` Return `true` if `item` ends with `substring`. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".ends_with("world") -> true ``` ## except `except(x, (v(, k)) => ...)` or `except(x, l)` Return a list or object where keys or entries matching the predicate have been removed. If the second argument is a lambda, it will be given the entry and if it returns `true`, the entry is removed. If the second argument is a list, any entry also found in this list will be removed. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "x-axis": 13.6, "y-axis": 63.1, "z-axis": 1.4, "offset": 4.3, "power": "on" }.except(["offset", "power"]) -> { "x-axis": 13.6, "y-axis": 63.1, "z-axis": 1.4 } ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "a": 1, "b": 2, "c": 3, "d": 4 }.except((v, k) => v > 2) -> { "a": 1, "b": 2 } ``` ## exp `exp(x)` Return e to the power of `x`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} exp(1) -> 2.718281828459045 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} exp(10) -> 22026.465794806718 ``` ## filter `filter(x, it => ...)` Remove any item from the list `x` where the lambda function returns `false` or `null`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].filter(item => item > 2) -> [3, 4] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} input.data.map(row => { "timestamp": to_unix_timestamp(row.StartTime, "%Y-%m-%dT%H:%M:%S"), "value": try_float(row.Value, null), "externalId": concat("prefix/", column.Name), "type": "datapoint", }).filter(dp => dp.value is "number") ``` ## flatmap `flatmap(x, it => ...)` Apply the lambda function to every item in the list `x` and flattens the result. For example, if the lambda function returns a list, the result of the `flatmap` will just be a list instead of a list of lists. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [[1, 2, 3], [2, 3, 4], [3, 4, 5]].flatmap(list => list.map(item => item + 1)) -> [2, 3, 4, 3, 4, 5, 4, 5, 6] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} input.sensorData.flatmap(timeseries => timeseries.values.map(datapoint => { "value": datapoint.value, "timestamp": to_unix_timestamp(datapoint.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": concat(timeseries.location, "/", timeseries.sensor), "type": "datapoint" }) ) ``` ## float `float(x)` Convert `x` into a floating point number if possible. If the conversion fails, the whole mapping will fail. Consider using [try\_float](#try_float) instead if you need error handling. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} float("6.1") -> 6.1 ``` ## floor `floor(x)` Return `x` rounded down to the nearest integer. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} floor(16.2) -> 16 ``` ## format\_timestamp `format_timestamp(x, f)` Convert the Unix timestamp `x` into a string representation based on the format `f`. The format is given using the table found [here](https://docs.rs/chrono/latest/chrono/format/strftime/index.html). **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} format_timestamp(1694159249120, "%Y-%m-%d %H:%M:%S") -> "2023-09-08 07:47:29" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} format_timestamp(now(), "%d/%m - %Y") -> "08/09 - 2023" ``` ## if `if(x, y, (z))` Return `y` if `x` evaluates to `true`, otherwise return `z`, or `null` if `z` is omitted. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} if(condition, "yes", "no") ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} if(true, "on", "off") -> "on" ``` ## if\_value `if_value(item, item => ...)` Map a value using a lambda if the value is not null. This is useful if you need to combine parts of some complex object or result of a longer calculation. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello".if_value(a => concat(a, " world")) -> "hello world" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} null.if_value(a => a + 1) -> null ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3].if_value(a => a[0] + a[1] + a[2]) -> 6 ``` ## int `int(x)` Convert `x` into an integer if possible. If the conversion fails, the whole mapping will fail. Consider using [try\_int](#try_int) instead if you need error handling. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} int("6") -> 6 ``` ## join `join(a, b, ...)` Return the union of the given objects or arrays. If a key is present in multiple objects, each instance of the key is overwritten by later objects. Arrays are simply merged. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} join({"key1": "value1"}, {"key2": "value2"}) -> { "key1": "value1", "key2": "value2" } ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} join([1, 2, 3], [4, 5], [6, 7, 8]) -> [1, 2, 3, 4, 5, 6, 7, 8] ``` ## length `length(x)` Return the length on the list, string or object `x`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} length("Hello, world") -> 12 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} length([1, 2, 3]) -> 3 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} length(input.items) ``` ## log `log(x, y)` Return the base `y` logarithm of `x`. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} log(16, 2) -> 4.0 ``` ## lower `lower(x)` Convert all characters in the string `x` to lowercase. If `x` is a boolean or number, it will be converted to a string. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "Hello World".lower() -> "hello world" ``` ## map `map(x, (it(, index)) => ...)` Apply the lambda function to every item in the list `x`. The lambda takes an optional second input which is the index of the item in the list. If applied to an object, the first input is the value, and the second is the key. The result is the new value. If the value is `null`, the lambda is ignored and `map` returns `null`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].map(number => number * 2) -> [2, 4, 6, 8] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} input.data.map(item => { "type": "datapoint", "value": item.value, "externalId": concat("prefix:", item.tag), "timestamp": now() }) ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ["a", "b", "c"].map((item, index) => index) -> [1, 2, 3] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} {"a": 1, "b": 2, "c": 3}.map((value, key) => concat(value, key)) -> {"a": "1a", "b": "2b", "c": "3c"} ``` ## max `max(a, b, ...)` Returns the larger of the given numbers. Can also be used on an array. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} max(1, 2) -> 2 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} max(1, 5, 2.0, 6) -> 6.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 8, 9, 2, 5, 4].max() -> 9 ``` ## min `min(a, b, ...)` Returns the smallest of the given numbers. It can also be called on an array. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} min(1, 2) -> 1 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} min(1, 5, 2.0, 6) -> 1.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 8, 9, 2, 5, 4].min() -> 1 ``` ## now `now()` Return the current time as a millisecond Unix timestamp, that is, the number of milliseconds since midnight 1/1/1970 UTC. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "timestamp": now() } ``` ## pairs `pairs(x)` Convert the object `x` into a list of key/value pairs. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "a": 1, "b": 2, "c": 3 }.pairs() -> [{ "key": "a", "value": 1 }, { "key": "b", "value": 2 }, { "key": "c", "value": 3 }] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "x-axis": 12.4, "y-axis": 17.3, "z-axis": 2.1 }.pairs().map(kv => { "timestamp": now(), "value": kv.value, "externalId": kv.key, "type": "datapoint" }) ``` ## parse\_json `parse_json(string)` Parse a string as a JSON object, which can be used in further transformations. If the passed value isn't a string, it's returned as-is. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} parse_json("{\"a\": 1, \"b\": 2}") -> {"a": 1, "b": 2} ``` ## pow `pow(x, y)` Return `x` to the power of `y` **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} pow(5, 3) -> 125.0 ``` ## random `random()` Return a random floating-point number between 0.0 (inclusive) and 1.0 (exclusive). **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} random() -> 0.123456789 ``` ## reduce `reduce(x, (acc, val) => ..., init)` Return the value obtained by reducing the list `x`. The lambda function is called once for each element in the list `val`, and the returned value is passed as `acc` in the next iteration. The `init` will be given as the initial `acc` for the first call to the lambda function. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4, 5].reduce((acc, val) => acc + val, 0) -> 15 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4, 5].reduce((acc, val) => acc * val, 1) -> 120 ``` ## regex\_all\_captures `regex_all_captures(haystack, regex)` Return an array of objects containing all capture groups from each match of the regex in the haystack. Unnamed capture groups are named after their index, so the match itself is always included as capture group `0`. If no match is found, this returns an empty array. See [regex\_is\_match](#regex_is_match) for details on regex support. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_all_captures("f123 f45 ff", "f(?[0-9]+)") -> [{ "0": "f123", "v": "123" }, { "0": "f45", "v": "45" }] ``` ## regex\_all\_matches `regex_all_matches(haystack, regex)` Return an array of all the substrings that match the regex. If no match is found, this returns an empty array. If you only need the first match, use [regex\_first\_match](#regex_first_match). See [regex\_is\_match](#regex_is_match) for details on regex support. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_all_matches("tests", "t[a-z]") -> ["te", "ts"] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_all_matches("foo bar baz", "\\w{3}") -> ["foo", "bar", "baz"] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_all_matches("test", "not test") -> [] ``` ## regex\_first\_captures `regex_first_captures(haystack, regex)` Return an object containing all capture groups from the first match of the regex in the haystack. Unnamed capture groups are named after their index, so the match itself is always included as capture group `0`. If no match is found, this returns null. See [regex\_is\_match](#regex_is_match) for details on regex support. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_first_captures("test foo bar", "test (?\\w{3}) (\\w{3})") -> { "0": "test foo bar", "v1": "foo", "2": "bar" } ``` ## regex\_first\_match `regex_first_match(haystack, regex)` Return the first substring in the haystack that matches the regex. If no match is found, this returns `null`. Prefer [regex\_is\_match](#regex_is_match) if all you need is to check for the existence of a match. See [regex\_is\_match](#regex_is_match) for details on regex support. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_first_match("test", "te") -> "te" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_first_match("te[st]{2}") -> "test" ``` ## regex\_is\_match `regex_is_match(haystack, regex)` Return `true` if the haystack matches the regex. Prefer this over the other regex methods if you only need to check for the presence of a match. The regex functions support a limited form of regex without certain complex features, such as backreferences and look-around. See [all the available regex syntax](https://docs.rs/regex/1.11.0/regex/index.html#syntax). We recommend using [regex101](https://regex101.com) with the mode set to `rust` for debugging regex. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_is_match("test", "te") -> true ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_is_match("test", "^not test$") -> false ``` ## regex\_replace `regex_replace(haystack, regex, replace)` Replace the first occurrence of the regex in the haystack. The replace object supports referencing capture groups using either the index (`$1`) or the name (`$group`). Use `$$` if you need a literal `$` symbol. `${group}` is equivalent to `$group` but lets you specify the group name exactly. See [regex\_is\_match](#regex_is_match) for details on regex support. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_replace("test", "te(?[st]{2})", "fa$v") -> "fast" ``` ## regex\_replace\_all `regex_replace_all(haystack, regex, replace)` Replace each occurrence of the regex in the haystack. See [regex\_replace](#regex_replace) for details. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} regex_replace_all("tests", "t(?[se])", "${v}t") -> etsst ``` ## replace `replace(a, b, c)` Replaces a string with another string **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "tomato".replace("tomato","potato") -> "potato" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} replace("potato","o","a") -> "patata" ``` ## round `round(x)` Return `x` rounded to the nearest integer. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} round(16.2) -> 16 ``` ## select `select(x, (v(, k)) => ...)` or `select(x, [1, 2, 3])` Return a list or object where the lambda returns true. If the second argument is a list, the list values or object keys found in that list are used to select from the source. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "x-axis": 13.6, "y-axis": 63.1, "z-axis": 1.4, "offset": 4.3, "power": "on" }.select(["x-axis", "y-axis", "z-axis"]) -> { "x-axis": 13.6, "y-axis": 63.1, "z-axis": 1.4 } ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "a": 1, "b": 2, "c": 3 }.select((v, k) => v > 2) -> { "c": 3 } ``` ## sin `sin(x)` Return the sine of `x`, where `x` is in radians. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sin(0) -> 0.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sin(3.141592653589793 / 2) -> 1.0 ``` ## slice `slice(x, start(, end))` Create a sub-array from an array `x` from `start` to `end`. If `end is not specified, go from `start`the end of the array. If`start`or`end\` are negative, count from the end of the array. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].slice(1, 3) -> [2, 3] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].slice(0, -3) -> [1] ``` ## split `split(a, b)` Split string `a` on any occurrences of `b`. If `b` is an empty string, this will split on each character, including before the first and after the last. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".split(" ") -> ["hello", "world"] ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello".split("") -> ["", "h", "e", "l", "l", "o", ""] ``` ## sqrt `sqrt(x)` Return the square root of `x`. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} sqrt(16) -> 4.0 ``` ## starts\_with `starts_with(item, substring)` Return `true` if `item` starts with `substring`. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".starts_with("hello") -> true ``` ## string `string(x)` Convert `x` into a string. `null`s will be converted into empty strings. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} string(true) -> "true" ``` ## string\_join `string_join(x(, a))` Return a string with all the elements of `x`, separated by `a`. If `a` is omitted, the strings will be joined without any separator. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ["hello", "there"].string_join(" ") -> "hello there" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3].string_join() -> "123" ``` ## substring `substring(x, start(, end))` Create a substring of an input string `x` from `start` to `end`. If `end` is not specified, go from `start` to end of string. If `start` or `end` are negative, count from the end of the string. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".substring(3, 8) -> "lo wo" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".substring(0, -3) -> "hello wo" ``` ## sum `sum(x)` Sum the numbers in the array `x`. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4].sum() -> 10 ``` ## tail `tail(x(, n))` Take the last element of the list `x`. If `n` is given, takes the last `n` elements, and returns a list if `n` > 1. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4, 5].tail() -> 5 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3, 4, 5].tail(2) -> [4, 5] ``` ## tan `tan(x)` Return the tangent of `x`, where `x` is in radians. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} tan(0) -> 0.0 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} tan(3.141592653589793 / 4) -> 1.0 ``` ## to\_object `to_object(x, val => ...(, val => ...))` Convert the array `x` into an object by producing the key and value from two lambdas. The first lambda produces the key, and the second (optional) produces the value. If the second is left out, the input is used as a value directly. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3].to_object(v => string(v + 1)) -> { "2": 1, "3": 2, "4": 3 } ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [1, 2, 3].to_object(v => string(v + 1), v => v - 1) -> { "2": 0, "3": 1, "4": 2 } ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} {"a": 1, "b": 2, "c": 3}.pairs().to_object(pair => pair.key, pair => pair.value) -> {"a": 1, "b": 2, "c": 3} ``` ## to\_unix\_timestamp `to_unix_timestamp(x, f)` Convert the string `x` into a millisecond Unix timestamp using the format string `f`. The format is given using the table found [here](https://docs.rs/chrono/latest/chrono/format/strftime/index.html). **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} to_unix_timestamp("2023-05-01 12:43:23", "%Y-%m-%d %H:%M:%S") -> 1682945003000 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "timestamp": to_unix_timestamp(input.time, "%Y-%m-%d %H:%M:%S") } ``` ## translate `translate(x, from, to)` Replace characters in the string `x` found in the string `from` with the corresponding character in the string `to`. If `to` and `from` are of different lengths, the expression will fail. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "hello world".translate("he", "HE") -> "HEllo world" ``` ## trim\_whitespace `trim_whitespace(x)` Remove any whitespace from the start and end of `x` **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} " hello ".trim_whitespace() -> "hello" ``` ## try\_bool `try_bool(a, b)` Try convert `a` to a boolean, if it fails, return `b`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_bool("true", null) -> true ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_bool("foo", null) -> null ``` ## try\_float `try_float(a, b)` Try convert `a` to a float, if it fails, return `b`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_float("6.2", 1.2) -> 6.2 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_float("4,5", null) -> 4.5 ``` ## try\_int `try_int(a, b)` Try convert `a` to a int, if it fails, return `b`. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_int("6", 1) -> 6 ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} try_int("4", null) -> 4 ``` ## upper `upper(x)` Convert all characters in the string `x` to uppercase. If `x` is a boolean or number, it will be converted to a string first. **Code examples** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "Hello World".upper() -> "HELLO WORLD" ``` ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} true.upper() -> "TRUE" ``` ## uuid4 `uuid4()` Generate a random UUID (version 4) and return it as a string. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} uuid4() -> "a3bb189e-8bf9-3888-9912-ace4e6543002" ``` ## zip `zip(x, y, ..., (i1, i2, ...) => ...)` Take a number of arrays, call the given lambda function on each entry, and return a single array from the result of each call. The returned array will be as long as the longest argument, null will be given for the shorter input arrays when they run out. **Code example** ```kuiper theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} zip([1, 2, 3], ["a", "b", "c"], (a, b) => concat(a, b)) -> ["1a", "2b", "3c"] ``` # Custom message formats Source: https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings Learn how to define custom message formats as mappings from source data to Cognite Data Fusion (CDF) data types, with examples for data points, events, and nested structures. A format can produce multiple different types of resources. The output must be a list of JSON objects or a single JSON object, where each matches an output resource schema described below. Note that the output can contain multiple different types, for example: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ { "type": "datapoint", "externalId": "my-ts", "timestamp": 1710842060124, "value": 123 }, { "type": "datapoint", "externalId": "my-ts", "timestamp": 1710842061124, "value": 124 }, { "type": "time_series", "externalId": "my-ts", "name": "My Time Series", "isString": false } ] ``` ## Data points Each output is a single CDF data point mapped to a CDF time series. If the time series doesn't exist, it will be created. * `type`: **Required**, set to **datapoint**. * `externalId`: The external ID of the time series to insert the data point into. * `space`: The name of the data modeling space to write the data points to. * `timestamp`: The timestamp of the data point is given as a millisecond Unix timestamp. * `value`: The data point value as a number or a string. This may also be `null`, but only if `status` is specified and a variant of `Bad`. * `status`: Data point status code *symbol* as defined in [the time series docs](/dev/concepts/reference/status_codes). * `isStep`: The value of `isStep` used when creating the time series for this datapoint if it does not already exist. ## Events Each output is a single CDF event. The old event will be updated if the provided event already exists and has an external ID. Null fields will be ignored when updating events. This is almost identical to the "Create event" payload to the Cognite API. See [the API docs on events](/20230101/events/create-events) for details. * `type`: **Required**, set to **event**. * `externalId`: **Required**, the external ID of the event. * `startTime`: The start time of the event, given as a millisecond Unix timestamp. * `endTime`: The end time of the event, given as a millisecond Unix timestamp. * `eventType`: The `type` field of the event in CDF. * `subtype`: The `subtype` field of the event in CDF. * `description`: The `description` field of the event in CDF. * `metadata`: An object containing key/value pairs for event metadata. Both keys and values must be strings. * `assetIds`: A list of asset **external ID**s to link the event to. If the assets do not exist, they will *not* be created. * `source`: The `source` field of the event in CDF. ## CDF RAW Each output is a single raw row, which will be ingested directly into CDF RAW. The extractor may skip ingesting raw rows that haven't changed since they were last seen. * `type`: **Required**, set to **raw\_row**. * `table`: The RAW table to write to. * `database`: The RAW database to write to. * `key`: The key of the row to be written to RAW. * `columns`: A JSON object containing the columns to be written to RAW. ## Time series Each output is a single time series. If the provided time series external ID already exists, then the old time series will be updated. Null fields will be ignored when updating events. This is almost identical to the "Create time series" payload to the Cognite API. See [the API docs on time series](/20230101/time-series/create-time-series) for details. * `type`: **Required**, set to **time\_series**. * `externalId`: **Required**, the external ID of the time series. * `name`: The name of the time series. * `isString`: Defines whether the time series contains string values (true) or numeric values (false). This value cannot be updated. Defaults to **false**. * `metadata`: An object containing key-value pairs for time series metadata. Both keys and values must be strings. * `unit`: The physical unit of the time series as free text. * `unitExternalId`: The physical unit of the time series as represented in the unit catalog. Only available for numeric time series. * `assetId`: Asset **external ID** to link the time series. If the asset does not exist, it will *not* be created. * `isStep`: Whether the time series is a step series or not. * `description`: Free text description of the time series. * `securityCategories`: List of security categories required to access the created time series. ## Data models Each output is a node or edge, written to a single view. Values are overwritten. Your mapping can produce a list of nodes or edges. If multiple outputs have the same external ID and space, these are combined. This lets you simultaneously ingest data into multiple views. See [the API docs](/api-reference/concepts/20230101/instances) for details. ### Nodes * `type`: **Required**, set to **node**. * `externalId`: **Required**, the external ID of the node. * `space`: **Required**, the space of the node. * `nodeType`: A direct relation to a different node representing the type of this node. * `externalId`: **Required**, type external ID. * `space`: **Required**, type space. * `view`: **Required**, the view being written to. * `externalId`: **Required**, view external ID. * `space`: **Required**, view space. * `version`: **Required**, view version. * `properties`: **Required**, a (maybe empty) object containing the properties being written, see [API docs](/api-reference/concepts/20230101/instances) for the string representation of data modeling data types. ### Edges * `type`: **Required**, set to **edge**. * `externalId`: **Required**, the external ID of the node. * `space`: **Required**, the space of the node. * `edgeType`: **Required**, a direct relation to a node representing the type of this edge. * `externalId`: **Required**, type external ID. * `space`: **Required**, type space. * `view`: **Required**, the view being written to. * `externalId`: **Required**, view external ID. * `space`: **Required**, view space. * `version`: **Required**, view version. * `properties`: **Required**, a (maybe empty) object containing the properties being written, see [API docs](/api-reference/concepts/20230101/instances) for the string representation of data modeling data types. * `startNode`: **Required**, the start node of the edge. * `externalId`: **Required**, start node external ID. * `space`: **Required**, start node space. * `endNode`: **Required**, the end node of the edge. * `externalId`: **Required**, end node external ID. * `space`: **Required**, end node space. ## Records Each output is a single immutable record written to a CDF stream. Records are append-only and time-ordered — once written, they cannot be modified. Writing to Records is a [public preview](/cdf/product_feature_status) feature. To opt in, set `beta` to `true`. * `type`: **Required**, set to **immutable\_record**. * `space`: **Required**, the space of the record. * `externalId`: **Required**, the external ID of the record. * `stream`: **Required**, the external ID of the stream to write to. * `sources`: **Required**, a list of container sources. Each source has: * `source`: **Required**, a reference to the container. * `type`: **Required**, set to **container**. * `space`: **Required**, the space of the container. * `externalId`: **Required**, the external ID of the container. * `properties`: **Required**, an object containing the properties to write, matching the container schema. * `beta`: **Required**, must be set to **true**. For a complete hosted extractor pipeline that writes to Records, see [Build a stream-to-stream transformation pipeline](/cdf/dm/records/guides/records_getting_started#advanced-build-a-stream-to-stream-transformation-pipeline). ## Example This section explains how you can set up your own message formats for hosted extractors in CDF using the MQTT extractor as an example. The language you must use to transform the data is inspired by JavaScript. If you want a complete overview and descriptions of all the available functions, see [All built-in functions](/cdf/integration/guides/extraction/hosted_extractors/built_in_functions). For more details about the mapping language, see [Mapping concepts](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts). ### Single data point in a message The section below describes how to set up message formats using a broker where the data comes in as values, and the topic indicates which sensor generated the value. For example, if you have a sensor that publishes the message ``` 23.5 ``` on the topic `/myhouse/groundfloor/livingroom/temperature` and you want to map this to a target in CDF. In this example, the data is ingested as a data point in a time series, where the content is the value, the time the message was received as the timestamp, and the topic path as the external ID of the time series. These are the input objects for defining the format: * `input`, which is the content of the message received * `context`, which contains information about the message, such as which topic it arrived on To make a data point from this message, create a JSON object with four fields: * `value`: The value of the data point. * `timestamp`: The timestamp for the data point given as a millisecond Unix timestamp. Use the built-in `now()` function to get the current time on this format. * `externalId`: The external ID of the time series to insert this data point into. * `type`: Set to `datapoint` to tell the extractor that this JSON object describes a data point. The final transform looks like: ``` { "value": input, "timestamp": now(), "externalId": context.topic, "type": "datapoint" } ``` ### Handling more data in a single message This section describes how to set up message formats while subscribing to a topic that contains the external ID of the time series getting the data, but the message payloads are now lists of data points instead of just a single value: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sensorData": [ { "datetime": "2023-06-13T14:52:34", "temperature": 21.4 }, { "datetime": "2023-06-13T14:59:53", "temperature": 22.1 }, { "datetime": "2023-06-13T15:23:42", "temperature": 24.0 } ] } ``` The content of messages will be automatically parsed as JSON and made available through the `input` object. In the last example, you used the `input` object directly, as the messages had no structure. This time, you can access attributes the same way you might be familiar with from object-oriented languages. For example `input.sensorData[0].temperature` will resolve to `21.4`. To ingest this data into CDF, you must make a data point for each element in the `sensorData` list. Use a `map` function on `input.sensorData`. `map` takes in a function and applies that function to each element in the list. ``` input.sensorData.map(row => ... ) ``` `row` is a name for the input of the `map` function. In this case, for the first iteration of the `map` the `row` object will look like ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "datetime": "2023-06-13T14:52:34", "temperature": 21.4 } ``` The output of the `map` function should be ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "value": 21.4, "timestamp": 1686667954000, "externalId": "/myhouse/groundfloor/livingroom/temperature", "type": "datapoint" } ``` To do this, you must define a JSON structure where you specify input data as the values: ``` { "value": row.temperature, "timestamp": to_unix_timestamp(row.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": context.topic, "type": "datapoint" } ``` For `value`, we map it to the `temperature` attribute of the `row` object. Similarly, for `timestamp`, except that we need to parse the time format from a string to a CDF timestamp. To do this, we use the `to_unix_timestamp` function, which takes in the timestamp to convert, and a description of the format. For the external ID of the time series to use, we do the same as in the previous example and use the topic the message arrived at. And `type` can just be hard coded to `datapoint` since we only ingest data points in this example. Putting that all together, we end up with the following format description: ``` input.sensorData.map(row => { "value": row.temperature, "timestamp": to_unix_timestamp(row.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": context.topic, "type": "datapoint" }) ``` ### Nested structures Finally, let's look at a case where the data is nested with several lists. For example, let's consider the case where a message contains a list of time series, each with a list of data points: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sensorData": [ { "sensor": "temperature", "location": "myhouse/groundfloor/livingroom", "values": [ { "datetime": "2023-06-13T14:52:34", "value": 21.4 }, { "datetime": "2023-06-13T14:59:53", "value": 22.1 }, { "datetime": "2023-06-13T15:23:42", "value": 24.0 } ] }, { "sensor": "pressure", "location": "myhouse/groundfloor/livingroom", "values": [ { "datetime": "2023-06-13T14:52:34", "value": 997.3 }, { "datetime": "2023-06-13T14:59:53", "value": 995.1 }, { "datetime": "2023-06-13T15:23:42", "value": 1012.8 } ] } ] } ``` First, let's start by iterating over the `sensorData` list in the same way as before: ``` input.sensorData.map(timeseries => ... ) ``` For the first iteration in this `map`, the `timeseries` object will then be ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sensor": "temperature", "location": "myhouse/groundfloor/livingroom", "values": [ { "datetime": "2023-06-13T14:52:34", "value": 21.4 }, { "datetime": "2023-06-13T14:59:53", "value": 22.1 }, { "datetime": "2023-06-13T15:23:42", "value": 24.0 } ] } ``` To extract the data points from this object, we need to iterate over the `values` list. Let's attempt to use `map` again to do that: ``` input.sensorData.map(timeseries => timeseries.values.map(datapoint => ... ) ) ``` For the first iteration of this inner `map`, the `datapoint` object will be ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "datetime": "2023-06-13T14:52:34", "value": 21.4 } ``` We can convert this to a data point JSON in a similar way to before: ``` { "value": datapoint.value, "timestamp": to_unix_timestamp(datapoint.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": concat(timeseries.location, "/", timeseries.sensor), "type": "datapoint" } ``` We also need to make an external ID for the time series ourselves. To do this, we use the `location` and `sensor` attributes on the `timeseries` object from the outer loop and join them together with the `concat` function. Notice that in this inner loop, both the `timeseries` object from the outer `map` and the `datapoint` object from the inner `map` are available. Putting this all together, we get ``` input.sensorData.map(timeseries => timeseries.values.map(datapoint => { "value": datapoint.value, "timestamp": to_unix_timestamp(datapoint.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": concat(timeseries.location, "/", timeseries.sensor), "type": "datapoint" }) ) ``` However, if we use this format to convert the example message, we will not get a list of data points, but a list of lists of data points: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ [ { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686667954000, "type": "datapoint", "value": 21.4 }, { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686668393000, "type": "datapoint", "value": 22.1 }, { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686669822000, "type": "datapoint", "value": 24 } ], [ { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686667954000, "type": "datapoint", "value": 997.3 }, { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686668393000, "type": "datapoint", "value": 995.1 }, { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686669822000, "type": "datapoint", "value": 1012.8 } ] ] ``` This is because `map` always works on a list and returns a new list. Since we want our output to be a list of data points, we need to change the outter `map` to a `flatmap`. `flatmap` is similar to `map`, except it *flattens* the output, which means that it rolls out the list of lists to just a simple list: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [ { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686667954000, "type": "datapoint", "value": 21.4 }, { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686668393000, "type": "datapoint", "value": 22.1 }, { "externalId": "myhouse/groundfloor/livingroom/temperature", "timestamp": 1686669822000, "type": "datapoint", "value": 24 }, { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686667954000, "type": "datapoint", "value": 997.3 }, { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686668393000, "type": "datapoint", "value": 995.1 }, { "externalId": "myhouse/groundfloor/livingroom/pressure", "timestamp": 1686669822000, "type": "datapoint", "value": 1012.8 } ] ``` In total, our final format looks like ``` input.sensorData.flatmap(timeseries => timeseries.values.map(datapoint => { "value": datapoint.value, "timestamp": to_unix_timestamp(datapoint.datetime, "%Y-%m-%dT%H:%M:%S"), "externalId": concat(timeseries.location, "/", timeseries.sensor), "type": "datapoint" }) ) ``` For more details about the mapping language, see [Mapping concepts](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts). ## Cookbook This section contains examples of common patterns in payloads, and mappings to handle them. ### Single data point with ID Each message is a single data point with ID: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "tag": "my-tag", "value": 123, "timestamp": "2023-06-13T14:52:34" } ``` This should be mapped to a single data point in CDF: ``` { "type": "datapoint", "timestamp": to_unix_timestamp(input.timestamp, "%Y-%m-%dT%H:%M:%S"), "value": input.value, "externalId": input.tag } ``` Sometimes, the value may be null, or some other value not accepted by the Cognite API. Mappings can return empty arrays, in which case nothing will be written to CDF: ``` [{ "type": "datapoint", "timestamp": to_unix_timestamp(input.timestamp, "%Y-%m-%dT%H:%M:%S"), "value": try_float(input.value, null), "externalId": input.tag }].filter(datapoint => datapoint.value is not null) ``` ### Multiple data points, single timestamp A relatively common pattern is that a data point contains several measurements with a single timestamp. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "sensorId": "my-sensor", "timestamp": "2023-06-13T14:52:34", "humidity": 123.456, "pressure": 321.654, "temperature": 15.1 } ``` Select these dynamically: ``` ["humidity", "pressure", "temperature"] .map(field => { "type": "datapoint", "timestamp": to_unix_timestamp(input.timestamp, "%Y-%m-%dT%H:%M:%S"), "value": try_float(input[field], null), "externalId": concat(input.sensorId, "/", field) }) .filter(datapoint => datapoint.value is not null) ``` If the actual fields are not known, you can even pick them dynamically using `pairs`: ``` input.pairs() .filter(pair => pair.key != "sensorId" && pair.key != "timestamp") .map(pair => { "type": "datapoint", "timestamp": to_unix_timestamp(input.timestamp, "%Y-%m-%dT%H:%M:%S"), "value": try_float(pair.value, null), "externalId": concat(input.sensorId, "/", pair.key) }) .filter(datapoint => datapoint.value is not null) ``` ### Sample data into a raw table Avoid doing this for very large data volumes. When developing a connection, it can sometimes be nice to get a sample of the data being ingested. You can write it to CDF Raw using a simple mapping like: ``` { "type": "raw_row", "table": "sample-table", "database": "sample-db", "key": string(now()), "columns": input } ``` # Message formats Source: https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings Complete reference for built-in and custom message payload formats supported by Cognite hosted extractors, including simple values, Cognite JSON, Sparkplug B, and Rockwell FactoryTalk. When a hosted extractor receives a message from a source, it parses the payload using a **message format** you configure on the extraction job. The format determines how the extractor interprets the raw bytes and maps them to Cognite Data Fusion (CDF) resources such as time series data points. Message formats are independent of the source type. You can use any format with any source: MQTT, Kafka, Event Hub, or REST. ## Built-in formats Built-in formats handle common payload structures without requiring you to write a mapping expression. Each built-in format produces time series data points. | Format | Payload type | Use case | | - | - | - | | [Simple value](#simple-value) | Plain text | Single sensor readings where the topic or key identifies the time series | | [Cognite](#cognite) | JSON | Systems that produce the Cognite JSON standard for time series ingestion | | [Sparkplug B](#sparkplug-b) | Protobuf | Industrial IoT systems that use the [Sparkplug B](https://sparkplug.eclipse.org/) specification | | [Rockwell FactoryTalk](#rockwell-factorytalk) | JSON | Rockwell FactoryTalk DataMosaix gateway data | All built-in formats share two optional configuration fields: * **External ID prefix**: A constant string prepended to every external ID the extractor generates. * **External ID from topic**: When enabled, the source topic or key is included in the external ID. If both **External ID prefix** and **External ID from topic** are set, the generated ID is in the form `[prefix][topic][id]`, where `id` is the field the format itself uses (for example, the metric name for Sparkplug B or the `tag_id` for Rockwell FactoryTalk). **External ID from topic** relies on the source providing a topic. MQTT and Kafka sources provide one; REST and Event Hub sources don't, so **External ID from topic** has no effect for those sources. ### Simple value Use this format when each message is a single sensor reading as plain text, and the topic or key identifies which sensor produced the value. The extractor uses the topic as the external ID of the time series, the message content as the value, and the current time as the timestamp. #### External ID construction The topic is used as the external ID. **External ID prefix** is prepended when configured. **External ID from topic** has no additional effect for this format because the topic is already the ID. #### Example message ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} 32.5 ``` ### Cognite The Cognite message format is a JSON standard for transmitting time series data to CDF. Messages must be UTF-8 encoded JSON. The structure mirrors the CDF API for familiarity. Each message contains a `timeseriesData` array. Each entry specifies an `externalId` and an `items` array of data points with `timestamp` (millisecond Unix epoch) and `value` (number or string) fields. You can optionally include a `status` field on each data point to set a [status code](/dev/concepts/reference/status_codes). #### External ID construction The `externalId` field of each `timeseriesData` entry is used as the external ID. When **External ID from topic** is enabled, the topic is prepended to it. #### Example message ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "timeseriesData": [ { "externalId": "my-timeseries", "items": [ { "timestamp": 1676550645000, "value": 1.6 }, { "timestamp": 1676550663020, "value": 5.6 } ] }, { "externalId": "my-string-timeseries", "items": [ { "timestamp": 1676550645000, "value": "on" }, { "timestamp": 1676550663020, "value": "off", "status": "UncertainSubNormal" } ] } ] } ``` ### Sparkplug B Use this format for sources that publish [Sparkplug B](https://sparkplug.eclipse.org/) payloads. Sparkplug B is a binary protocol built on protobuf, widely used in industrial IoT for reporting device metrics over MQTT. The extractor decodes each metric in the protobuf payload and converts it to a CDF data point. The following metric value types are supported: | Sparkplug type | Protobuf type | CDF value type | | - | - | - | | Int | `uint32` | Numeric | | Long | `uint64` | Numeric | | Float | `float` | Numeric | | Double | `double` | Numeric | | Boolean | `bool` | String (`"true"` / `"false"`) | | String | `string` | String (truncated to 255 characters) | Bytes, Dataset, Template, and Extension metric types are not supported and are skipped during ingestion. #### Payload structure Sparkplug B payloads are binary-encoded protobuf messages. Each payload contains a list of metrics, where each metric carries a name, a typed value, and an optional timestamp. The structure follows the [Sparkplug specification](https://sparkplug.eclipse.org/specification/): ```protobuf theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} message Payload { optional uint64 timestamp = 1; // Payload-level timestamp (ms since epoch) repeated Metric metrics = 2; // List of metrics optional uint64 seq = 3; // Sequence number } message Metric { optional string name = 1; // Metric name (e.g., "Motor/Temperature") optional uint64 timestamp = 3; // Metric-level timestamp (ms since epoch) oneof value { uint32 int_value = 10; uint64 long_value = 11; float float_value = 12; double double_value = 13; bool boolean_value = 14; string string_value = 15; // bytes, dataset, template, extension - not supported } } ``` #### Timestamp resolution The extractor uses the metric-level timestamp if present, then falls back to the payload-level timestamp, and finally to the current UTC time. Timestamps are in milliseconds since epoch. #### External ID construction The metric name is used as the external ID. When **External ID from topic** is enabled, the topic is prepended to the metric name. Sparkplug reserves several metric names for protocol control (such as `Node Control/Reboot`, `Node Control/Rebirth`, and `bdSeq`). The extractor automatically filters out these metric names and does not ingest them as data points. ### Rockwell FactoryTalk Use this format for data from [Rockwell FactoryTalk DataMosaix](https://www.rockwellautomation.com/en-us/products/software/factorytalk/datamosaix.html) gateways. The extractor expects a JSON payload with a `gatewayData` array, where each entry contains a `tag_id` and an array of value-quality-timestamp (`vqts`) objects. #### External ID construction The `tag_id` field is used as the external ID. When **External ID from topic** is enabled, the topic is prepended to it. #### Example message ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "gatewayData": [ { "tag_id": "Plant1/Reactor/Temperature", "vqts": [ { "v": 87.3, "t": "2024-01-15T10:30:00.000Z" }, { "v": "88.1", "t": "2024-01-15T10:31:00.000Z" } ] } ] } ``` #### Field mapping | Field | Description | | - | - | | `tag_id` | Used as the external ID of the time series. | | `v` | The data point value. Accepts a JSON number or a string representation of a number. | | `t` | Timestamp in [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format (for example, `2024-01-15T10:30:00.000Z`). | The payload may include additional fields such as `q` (quality), `model_id`, and `mimeType`. These fields are ignored during ingestion. Each VQT entry becomes a separate data point in CDF. ## Custom formats When your payload structure doesn't match a built-in format, define a **custom format** using mapping expressions. Custom formats let you parse the payload, extract fields, and map them to any supported CDF resource type. Supported resource types include data points, events, RAW rows, time series metadata, data model nodes and edges, and stream records. Custom formats also support additional input parsers beyond JSON: you can parse **CSV**, **XML**, and **protobuf** payloads before applying your mapping expression. For a complete guide to writing custom formats, see [Custom message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). ## Preprocessing Before the extractor applies a message format, it can run optional preprocessing steps on the raw payload bytes. These steps run in order: character encoding conversion first, then decompression. ### Compression If your source system sends compressed payloads, enable **GZip** decompression. The extractor decompresses the payload before passing it to the message format. ### Character encoding By default, the extractor expects UTF-8 encoded payloads. If your source system uses a different encoding, configure one of the following: | Encoding | Description | | - | - | | UTF-16 BE | UTF-16, big-endian byte order | | UTF-16 LE | UTF-16, little-endian byte order | | Latin-1 | Windows CP-1252 (superset of ISO 8859-1) | The extractor converts the payload to UTF-8 before parsing. ## Further reading * [Custom message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings) * [Hosted extractors](/cdf/integration/guides/extraction/hosted_extractors) # Mapping concepts Source: https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts Learn about the hosted extractor mapping language for transforming JSON structures with functional expressions similar to JavaScript. The hosted extractor mapping language translates JSON structures to other JSON structures. The language is similar to JavaScript and is purely *functional*. A mapping is a single expression that describes how to build one JSON structure from another. ## Variables Use these input objects to define a mapping: * `input` - This is the content of the received message, parsed into an object. * `context` - This contains information about the message. The content of this object varies between the different hosted extractors. For MQTT, the `context` object contains * `topic`: which topic the message arrived at For examples of using these variables, see [Handling more data in a single message](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings#handling-more-data-in-a-single-message). When variables are objects or arrays they can be selected from. For example, given the object ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "someField": 123, "someArray": [1, 2, 3, { "nested": [1, 2, 3] }] } ``` you can select from an object with ``` input.someField -> 123 ``` or ``` input["someField"] -> 123 ``` You select from arrays with ``` input.someArray[0] -> 1 ``` and from the end of arrays with ``` input.someArray[-2] -> 3 ``` Selectors can be nested as ``` input.someArray[3].nested[2] -> 3 ``` Selectors can contain expressions ``` input.someArray[1 + 1] -> 3 ``` ## Functions The hosted extractor mapping language contains a set of predefined functions, which can be used to transform the input in other ways. For example, ``` float("1.4") -> 1.4 ``` will try to transform the input from a string to a number, which may fail. Any function can also be called *postfix*, as a method on a value, so you could instead write ``` "1.4".float() -> 1.4 ``` This is especially useful when dealing with chained operations. See also [Functions](/cdf/integration/guides/extraction/hosted_extractors/built_in_functions). ## Lambda expressions Some functions can take user defined functions as arguments, such as `map`, `reduce`, or `filter`. To create the functions for these, use *Lambda notation*: `argument => return`. If the function takes no arguments, use `()` as the input to the function. If a function takes multiple values, denote them as a tuple: `(arg1, arg2)`. ### Examples If we want to turn the list `[1, 2, 3, 4]` into `[2, 4, 6, 8]` we can use `map` to apply a function to each member of the list, and write a lambda function that doubles each value: ``` [1, 2, 3, 4].map(number => number * 2) -> [2, 4, 6, 8] ``` If you want to remove all `null`s from a list, use the `filter` function, and write a lambda function that returns true if the value is the `null` type using the `is` operator: ``` [0, 1, 2, 3, 4, null, 5].filter(item => !(item is "null")) -> [0, 1, 2, 3, 4, 5] ``` To sum all the numbers from 1 through 10, use the `reduce` function to iteratively add the next member of the list to an aggregate: ``` [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].reduce((sum, next) => sum + next, 0) -> 110 ``` ## Operators The operators take one or two inputs and output some value. The operators are: ### Plus `1 + 2 + 3 -> 6` Adds two numbers together. ### Minus `1 - 2 - 3 -> -4` Subtracts a number from another. ### Multiply `2 * 3 -> 6` Multiplies two numbers. ### Divide `3 / 2 -> 1.5` Divides one number by another. ### And `true && false -> false` Boolean AND, returns `true` if the inputs are both not `null` or `false`. ### Or `true || false -> true` Boolean OR, returns `true` if either input is not `null` or `false`. ### Equals `1 == 2 -> false` Checks for exact equality. The inputs must be the same type, or this will be `false`. ### Not equals `1 != 2 -> true` Checks for inequality. ### Greater than `2 > 3 -> false` Checks if the first number is greater than the second number. ### Greater than or equal `2 >= 3 -> false` Checks if the first number is greater than or equal to the second number. ### Less than `2 < 3 -> true` Checks if the first number is less than the second number. ### Less than `2 <= 3 -> true` Checks if the first number is less than or equal to the second number. ### Modulo `5 % 2 -> 1` Computes the modulus, or remainder, of the first argument divided by the second argument. ### Is `5 is float` This is the type check operator, used to check if the first argument is the type given by the second argument. The valid types are * `null` * `object` * `array` * `string` * `number` * `float` * `int` * `bool` Any `float` or `int` is also `number`. ## If expressions You can use functional-style `if` expressions. For example ``` if 2 > 3 { "hello" } else if 2 == 2 { "world" } else { "goodbye" } ``` will return `"world"`. There is an `if` function as well, but using expressions usually results in cleaner code. ## Objects You can create JSON objects with familiar JSON syntax: ``` { "hello": 123, "test": input.something } ``` These objects can be accessed, just like input variables: `{ "hello": 123 }.hello` You can concat objects using the `...` operator: ``` { "test": 123, ...{ "hello": "world" }, ...{ "world": "hello" } } ``` will yield the object ``` { "test": 123, "hello": "world", "world": "hello" } ``` Object keys can also be expressions: `{ concat("he", "llo"): "world" }` will yield an object `{ "hello": "world" }` ## Arrays Arrays can be created using a JSON syntax: `[1, 2, 3]` You can access arrays using brackets, just like with inputs: `[1, 2, 3][0] -> 1` Arrays also support the `...` operator: ``` [ 1, ...[2, 3], ...[4, 5] ] ``` will yield `[1, 2, 3, 4, 5]` ## Macros Define your own reusable macros at the top of your mapping file: ``` #my_macro := (a, b) => a + b; // Then use it later in the mapping expression my_macro(3, 4) -> 7 ``` Macros may call each other, but may not form loops. ``` #my_macro := (a, b) => my_macro(1, 2); // Illegal #my_macro_1 := a => a + 1; #my_macro_2 := b => my_macro_1(b) + 1; // Legal ``` Avoid using the same argument name in macros that call each other. In the example above, it wouldn't be allowed to have `my_macro_1` take an argument named `b`. ## Variables You can define reusable intermediate variables at the top of lambdas or at the top of the expression: ``` #thing := 1 + 1; #other_thing := floor(thing); [1, 2, 3].map(a => #intermediary := a + other_thing; intermediary ) ``` Variables may not overlap with each other or with lambda variables. ## Template strings Template strings allow you to easily construct dynamic strings, instead of having to rely on the `concat` method: ``` $"Math: 1 + 1 = {1 + 1}, {'nested string'}" ``` would result in `"Math: 1 + 1 = 2, nested string"`. You can escape `{` and `}` using `{{` and `}}`, for example: ``` $"Hello: {{World}}" ``` would produce `"Hello {World}"`. # Kafka extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/kafka Learn how to use the Cognite Kafka extractor to extract data from Apache Kafka brokers and topics. Cognite hosts the Kafka extractor, so you don't have to download or install anything. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#cognite-kafka-extractor) to *create* a hosted Kafka extractor and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models in the target CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). ## Deploy the extractor Navigate to Data fusion > Integrate > Extractors. Locate the **Cognite Kafka extractor** and select **Set up extractor**. Kafka is highly redundant, so when you connect to a Kafka broker, you typically connect to one or more from a list of *bootstrap brokers*. This tells the client which specific broker to connect to. You provide this list of bootstrap brokers to the extractor. ## Message formats Kafka is an event streaming platform from Apache that supports payloads on any format. The Cognite Kafka extractor supports several [pre-defined message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings). If you want to define your own custom mapping of Kafka messages, see [custom data formats for hosted extractors](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). Custom formats used with Kafka jobs will receive an `input` argument containing the message as JSON, and a `context` argument containing the topic and some other message metadata, for example: ``` { "key": [1, 2, 3, 4], "topic": "stream_a", "headers": { "headerA": "valueA", "headerB": "valueB" }, "timestamp": 1710312447698, "current_offset": 5912, "partition_id": 1 } ``` Messages are published on *topics*. A Kafka broker has a fixed set of topics, each writing data to one or more *partitions*. When you create a Kafka extractor instance, you must specify the *number* of partitions and the *name* of the topic. This information can be used to help construct data points or other supported output types. # MQTT extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/mqtt Learn how to use the Cognite MQTT extractor to bring MQTT data into Cognite Data Fusion (CDF), including Quality of Service (QoS), broker persistence, and verifying delivery with hosted extractor job metrics. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#cognite-mqtt-extractor) to *create* a hosted MQTT extractor and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models in the target CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. For more information, see [setup and administration for extractors](/cdf/integration/guides/extraction/admin_oidc). ## Deploy the extractor Navigate to Data fusion > Integrate > Extractors. Locate the **Cognite MQTT extractor** and select **Set up extractor**. ## Message delivery and reliability Before you define message formats and mappings, review how data reaches CDF. Reliable delivery depends on your **publishers**, your **broker**, and the **Cognite MQTT extractor**, not on the extractor alone. The extractor subscribes to topics at **Quality of Service level 1**, which provides at-least-once delivery, and requests persistent sessions from the broker. For reliable message delivery during brief disconnections, your broker must persist messages for disconnected clients and retain session state. When the extractor reconnects, the broker can deliver unacknowledged messages. The broker applies the **lower** of the publisher's QoS and the extractor's subscription QoS. **QoS 0** publishes reach the extractor at QoS 0 only, even though the extractor subscribes at **QoS 1**. Configure your publishers to use **QoS 1** when sending messages, or **QoS 2** if your broker supports it. On your broker, turn on session persistence and offline message buffering according to your vendor's documentation. To verify that data reaches CDF, use the job metrics in [Hosted extractors overview](/cdf/integration/guides/extraction/hosted_extractors). The Cognite MQTT extractor does not guarantee zero message loss if network interruptions coincide with Cognite-hosted service restarts. If the broker shows traffic but CDF does not update, work through the checks below. **Symptoms:** The broker or your publisher logs show traffic, but time series, events, or other CDF resources do not update. **Checks:** 1. Confirm publisher **QoS**, broker persistence, and sessions as described earlier in this section. 2. In the [Hosted extractors overview](/cdf/integration/guides/extraction/hosted_extractors), open the job and compare **Source messages** with **Destination uploaded values** and related metrics to see whether failures happen at transform or write steps. 3. If you use an **extraction pipeline** to track ingestion, see [Monitor extraction pipelines](/cdf/integration/guides/interfaces/monitor_integrations#understanding-status-and-uptime) for how to interpret **Topic filters status** and uptime, and troubleshoot failed integrations. The same article explains troubleshooting for on-premises and hosted extractors. ## Message formats MQTT is a generic message transport protocol agnostic to the message payloads, which can be in any format. The Cognite MQTT extractor supports several [pre-defined message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings). If you want to define your own custom mapping of MQTT messages, see [custom data formats for hosted extractors](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). Custom formats used with MQTT jobs will receive an `input` argument containing the message as JSON, and a `context` argument containing the topic, for example: ``` { "topic": "company/location3/machine/pressure" } ``` The topic given in the context will be the *actual* topic the message arrived on, not the topic filter, so it can be used to distinguish identical messages arriving on different topics. Messages are published on *topics*. A topic can be any string of text, but usually it's a hierarchical structure, separated by forward slashes, for example: ``` company/location3/machine1/pressure ``` Clients can subscribe to messages by subscribing to *topic filters*. A topic filter can be a topic written verbatim, but it also allows for wildcards such that one subscription can yield data from multiple topics. MQTT supports single-level and multi-level wildcards. `+` denotes a single-level wildcard and matches any text between two slashes. For example, if you have several topics like these: ``` company/location3/machine1/pressure company/location3/machine2/pressure company/location3/machine3/pressure ``` You can subscribe to all of them by subscribing to this topic filter: ``` company/location3/+/pressure ``` `#` denotes a multi-level wildcard and matches any text. It has to be used at the end of the topic filter. For example, if you have several topics like the following: ``` company/location3/machine1/pressure company/location3/machine2/pressure company/location3/machine1/temperature ``` You can subscribe to all of them by subscribing to this topic filter: ``` company/location3# ``` ## Encryption The Cognite MQTT extractor supports connecting to MQTT brokers with Transport Layer Security (TLS) encryption. Select **Enable Transport Layer Security (TLS) on the connection** when you set up the connection to the broker. Upload the root certificate to the Cognite MQTT extractor when you're creating the connection if your broker uses a custom Certificate Authority (CA) to create and sign the certificate used for encryption. The extractor supports PEM and DER formats. **Certificate version** The Cognite MQTT extractor only supports X.509 certificates of version 3. Earlier versions will not work with the extractor. # MQTT broker Source: https://docs.cognite.com/cdf/integration/guides/extraction/mqtt_broker Learn how to publish data directly to a Cognite-managed MQTT broker and ingest it into Cognite Data Fusion (CDF). The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. The Cognite MQTT broker is an **inbound** endpoint — your systems publish data directly to a Cognite-managed MQTT broker. This is the reverse of the [Cognite MQTT extractor](/cdf/integration/guides/extraction/mqtt), where Cognite connects as a subscriber to your broker. Use the MQTT broker when your systems can push data over MQTT but you don't want to run and expose your own broker. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#cognite-mqtt-extractor) to *create* an MQTT broker source and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models to the target CDF project, depending on the resource types you want the extractor to ingest. * Before you create the job, reuse or create a [destination](/api-reference/concepts/20230101/hosted-extractors-destinations). If you use a custom message format, reuse or create a [mapping](/api-reference/concepts/20230101/hosted-extractors-mappings) as described in [Custom message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). * The MQTT broker must be enabled for your project. Contact Cognite support to request access. * The Cognite MQTT broker is not available for Private Link–enabled CDF projects. This setup uses two different credentials: * **CDF API** (create the source and job): use your usual CDF authentication, such as OpenID Connect (OIDC), as described in [setup and administration for extractors](/cdf/integration/guides/extraction/admin_oidc). * **MQTT publisher** (send data): use the `host`, `username`, and `password` returned when you create the source. ## Set up the MQTT broker Create a `Source` by sending a request to `/api/v1/projects/{project}/hostedextractors/sources`. Include the `cdf-version: beta` header. For the full request schema, see the [Source API specification](/api-reference/concepts/20230101/hosted-extractors-sources). ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [{ "type": "mqtt_broker", "externalId": "my-mqtt-broker-source" }] } ``` The response contains the connection details your publisher needs: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [{ "type": "mqtt_broker", "externalId": "my-mqtt-broker-source", "host": "", "port": 8883, "authentication": { "type": "basic", "username": "", "password": "" } }] } ``` The password is returned only when you create the source. Save the password by storing it immediately. If you read the source again later, the API will not return the password. If you lose the password, reset it as described in [Managing credentials](#managing-credentials). Create a `Job` by sending a request to `/api/v1/projects/{project}/hostedextractors/jobs`. The job must reference your destination and mapping. For the full request schema, see the [Job API specification](/api-reference/concepts/20230101/hosted-extractors-jobs). Set `topicFilter` to match the topics your publisher will send data on. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [{ "externalId": "my-mqtt-broker-job", "sourceId": "my-mqtt-broker-source", "format": { "type": "custom", "mappingId": "my-custom-mapping" }, "destinationId": "my-destination", "config": { "topicFilter": "company/+/sensors/#" } }] } ``` Messages must be transformed into a CDF resource type. For supported formats and custom mappings, see [Message formats](#message-formats). A `200` response with `connected` or `running` in the response body confirms that you created the job. If the [job](/cdf/integration/guides/extraction/hosted_extractors#jobs) is not `connected` or `running`, work through the checks below. You may see one of these responses: * `200` with `connection_error` or `startup_error` in the body * `400 Bad Request` or `422 Unprocessable Entity` Use the [Job API specification](/api-reference/concepts/20230101/hosted-extractors-jobs) to identify, decode, and resolve specific error codes. Also verify the following. * Your authentication token is valid. * Your request payload matches the request schema. Configure your MQTT client to connect using the credentials from step 1: * **Host**: the `host` value from the response in step 1 * **Port**: `8883` * **Username**: the `authentication.username` value * **Password**: the `authentication.password` value * **TLS**: required. See [Encryption](#encryption) for certificate requirements. A success message or response state in your MQTT client confirms that your publisher connected to the broker. If your publisher does not connect, work through the checks below. Your MQTT client may show an error message or a failed connection state. Use your MQTT client documentation to identify, decode, and resolve specific error codes, then try connecting again. Also verify the following. * The host, port, username, and password match the values from step 1. * TLS is enabled and your device trusts the broker certificate. See [Encryption](#encryption). * You saved the password from step 1 and have not reset it since. ## Managing credentials The password can be reset at any time by sending a `POST` request to `/api/v1/projects/{project}/hostedextractors/sources/update` with the `cdf-version: beta` header: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [{ "type": "mqtt_broker", "externalId": "my-mqtt-broker-source", "update": { "password": { "reset": true } } }] } ``` The response contains the new password. The previous password is immediately invalidated — connected publishers will be disconnected and must reconnect with the new credentials. ## Topics and topic filters Messages are published to **topics** — hierarchical strings separated by forward slashes: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} company/location3/machine1/pressure ``` Jobs listen for messages using *topic filters*. A topic filter can be an exact topic or a wildcard pattern, allowing a single job to receive data from multiple topics. MQTT supports single-level (`+`) and multi-level (`#`) wildcards. * `+` matches a single level. `company/+/pressure` matches `company/machine1/pressure` and `company/machine2/pressure`. * `#` matches all remaining levels and must appear at the end. `company/location3/#` matches all topics under `company/location3/`. Custom mappings receive the arrived topic in `context.topic`. See [Message formats](#message-formats) for examples of `input` and `context` arguments. ## Message formats The Cognite MQTT broker supports [pre-defined message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings) and [custom message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). Custom mappings receive two arguments: `input`, which contains the message payload as JSON, and `context`, which contains the topic the message arrived on. For example, when a publisher sends `{"pressure": 42.1}` on topic `company/location3/machine1/pressure`, the mapping receives the following values. **input** (message payload): ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "pressure": 42.1 } ``` **context** (arrival metadata): ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "topic": "company/location3/machine1/pressure" } ``` The topic in `context` is the actual topic the message arrived on, not the topic filter. Use `context.topic` to distinguish messages arriving on different topics. ## Encryption All connections to the Cognite MQTT broker require Transport Layer Security (TLS). The broker certificate is signed by [Let's Encrypt](https://letsencrypt.org/certificates/). Let's Encrypt root certificates are included in most modern operating systems — make sure the device running your MQTT client has up-to-date certificates. # About the Cognite OPC Classic extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_classic Read time series and asset information via OPC DA and HDA protocols and stream datapoints to Cognite Data Fusion (CDF). The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. The extractor copies the OPC hierarchy to CDF, then streams datapoints to time series in CDF. The OPC Classic extractor supports the following: * **Connecting to multiple OPC servers** over DCOM or COM+. * **Extracting string and numeric datapoints** from OPC tags. * **Reading historical data** from OPC HDA, storing progress in a local or remote state store. * **Starting from a local JSON file** instead of reading the server on each startup, improving performance. # Command-line arguments Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_classic/opc_classic_cli Available command-line arguments for the OPC Classic extractor. | Option | Description | | - | - | | --config-file\|-f | Override location of the YAML configuration file. This can be a relative or absolute path. | | --log-level\|-l | Set the console log level. Valid options are `verbose`, `debug`, `information`, `warning`, and `error`. | | --working-dir\|-w | Set the working directory of the extractor. The default value is the current directory for standalone, or one level above the executable file for the service version. | | --log-dir | Set the path to log files to enable logging to file. | | --service\|-s | Required when starting the extractor as a service. This option won't start the extractor as a service on its own, but makes the necessary modifications when it is called by Windows. | | --version | Show the current extractor version. | | -?\|--h\|--help | Show the available command line options. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_classic/opc_classic_configuration Configure the OPC Classic extractor using YAML format with settings for source, subscriptions, and CDF connection. To configure the OPC Classic extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org) format and the sample configuration file contains all valid options with default values. When setting up an extractor, you should *not* base your config on the file `config.example.yml`, but instead, use the `config.minimal.yml` as your base and copy the parts you need from `config.example.yml`. You can exclude fields entirely to let the extractor use default values. The configuration file separates settings by component, and you can remove an entire component to disable it or use default values. ## Environment variable substitution In the config file, values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ### Minimal YAML configuration file ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # Windows username for authentication username: # Windows password for authentication password: # List of servers to connect to. servers: - # Server host name or IP address host: # Version of DA to use, one of V2 or V3 # This can be left out to disable live data. da-version: # Version of HDA to connect to on this host. # This can be left out to disable history. # Must be V1 hda-version: # Prefix on externalIds for nodes generated by this server. id-prefix: # Name of state store used to store states if history is enabled for this server. Required if hda-version is set. state-store-name: endpoint-url: "opc.tcp://localhost:4840" cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: "${COGNITE_PROJECT}" # If this is set to true, credentials can be left out, and the extractor # will read data without pushing it to CDF. debug: false # This is for Microsoft as IdP, to use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} ``` ##
    Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`. For example, `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use cron expressions. For history start and end times, you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. ## Source This section contains parameters for connecting to the OPC Classic servers. The extractor can connect to multiple servers, where each server has its own ID prefix. Technically, each server may have *multiple* connections because DA and HDA are effectively separate servers. In practice, many servers support both DA and HDA interfaces but share information between the two. All servers share the same authentication information. This is under the assumption that the extractor signs in to the servers using a shared domain or similar. It is considered best practice to give the extractor a separate network user. Run multiple extractors if the extractor needs multiple sets of credentials. | Parameter | Description | | - | - | | **username** | Windows username to use for authentication to the servers. | | **password** | Windows password to use for authentication to the servers. | | **domain** | Domain for the user used for authentication. | | **parallelism** | Maximum number of requests made in parallel to each server. The default value is `10`. | | **use-async** | Use async mode when making requests to HDA servers. This can be more efficient both for the extractor and the server, but not all servers support it, so it's disabled by default. | | **servers** | A list of servers to extract from. | | **servers\[].host** | Host or IP address to connect to. | | **servers\[].da-version** | Version of DA to connect to on this host. This can be left out to not connect to DA at all. Valid options are `V2` or `V3`. | | **servers\[].hda-version** | Version of HDA to connect to. This can be left out to not connect to HDA at all. Must be set equal to `V1` if enabled. | | **servers\[].name** | Name of the server to connect to on the given host. This is used to pick which server to connect to if multiple are available. If left out, pick the first server found. | | **servers\[].proxy-address** | Proxy requests to the server through this address. | | **servers\[].id-prefix** | Prefix for external IDs of assets and time series created in CDF by this server. | | **servers\[].state-store-name** | Name of state store used to store states if history is enabled for this server. Required if **hda-version** is set. | | **servers\[].cache.path** | Path to a local JSON file caching the node hierarchy. If the file doesn't exist, the extractor will browse the DA and HDA servers and generate it. The file may be manually edited to limit which nodes the extractor should read. | | **attribute-chunking** | Configuration for chunking of attributes read from HDA servers. | | **attribute-chunking.chunk-size** | Maximum number of items per attribute read request. The default value is `1000`. | | **attribute-chunking.parallelism** | Maximum number of parallel requests for attributes. The default value is `10`. | | **keep-alive-interval** | Interval between each read of server status, used as a keep alive. The syntax is described in [Timestamps and intervals](#timestamps). | ## Subscriptions If you connect the extractor to a Data Access (DA) server, it establishes subscriptions on the tags it discovers. Subscriptions in OPC DA are callbacks, meaning that the server will call a function in the extractor whenever it sees any changes. | Parameter | Description | | - | - | | **chunking** | Configuration for how the extractor will chunk requests for subscriptions. | | **chunking.chunk-size** | Maximum number of items per subscription request. The default value is `1000`. | | **chunking.parallelism** | Maximum number of parallel requests to create subscriptions. The default value is `10`. | | **keep-alive** | Keep alive rate in milliseconds for subscriptions. The default value is `10000`. | | **update-rate** | Requested update rate, this is how often the server should check for updates from its underlying systems. The server is not required to obey this, and may return a revised update rate, or just use a notification based approach. The default value is `1000` | | **deadband** | Minimum difference in value required for an update to be registered. The default value is `0.0` | ## Logger Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, `Verbose`, in order of decreasing importance. Each level covers the ones of higher importance. | Parameter | Description | | - | - | | **console** | Configuration for logging to the console. | | **console.level** | Minimum level of log events to write to the console. Set this to enable console logging. | | **console.stderr-level** | Log events at this level or above are redirected to standard error. | | **file** | Configuration for logging to a rotating log file. | | **file.level** | Minimum level of log events to write to file. | | **file.path** | Path to the files to be logged. If this is, for example, set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | **file.retention-limit** | Maximum number of log files that are kept in the log folder. | | **file.rolling-interval** | A rolling interval for log files. Either `day` or `hour`. The default value is `day`. | ## Metrics The OPC Classic extractor can push some general metrics about usage to a Prometheus pushgateway server, or expose a prometheus server for scraping. | Parameter | Description | | - | - | | **server** | Configuration for a prometheus scrape server. | | **server.host** | Host for a locally hosted prometheus server, used for scraping. | | **server.port** | The port used by the local prometheus server. | | **push-gateways** | A list of pushgateway destinations the extractor will push metrics to. | | **push-gateways\[].host** | URI of the pushgateway host. | | **push-gateways\[].job** | Name of the metrics job on this pushgateway. | | **push-gateways\[].username** | Username for basic authentication. | | **push-gateways\[].password** | Password for basic authentication. | | **push-gateways\[].push-interval** | Interval in seconds between each push of metrics. | ## Cognite Configuration for the connection to Cognite Data Fusion (CDF). | Parameter | Description | | - | - | | **host** | The CDF service URL. Defaults to `https://api.cognitedata.com`. | | **project** | The CDF project. **Required**. | | **idp-authentication** | Configuration for authenticating to CDF. | | **idp-authentication.authority** | Authority used with `tenant` to authenticate to azure tenants. Use `token-url` if connecting using a non-azure IdP. Defaults to `https://login.microsoftonline.com` | | **idp-authentication.tenant** | Azure tenant used with `authority`. | | **idp-authentication.token-url** | URL used to obtain service tokens, used for non-azure IdPs. | | **idp-authentication.client-id** | Service principal client ID. | | **idp-authentication.secret** | Service principal client secret. | | **idp-authentication.resource** | Optional resource parameter to pass along with token request. | | **idp-authentication.scopes** | A list of scopes to pass along with the request, will typically need to contain `[host]/.default` | | **idp-authentication.audience** | Optional audience parameter to pass along with token request. | | **idp-authentication.min-ttl** | Requested minimum time-to-live in seconds for the token. | | **idp-authentication.certificate** | Configuration for authenticating using a client certificate. | | **idp-authentication.certificate.authority-url** | Certificate authority URL. | | **idp-authentication.certificate.path** | Path to the `.pem` or `.pfx` certificate to be used for authentication. | | **idp-authentication.certificate.password** | Certificate password. | | **data-set** | Configuration for data set to assign newly created assets and time series to. | | **data-set.id** | Internal ID of dataset. Specify either this or `external-id`. | | **data-set.external-id** | External ID of dataset. Specify either this or `id`. | | **update** | Set this to `true` to enable updating assets and time series that have changed in the source. | | **metadata-targets** | Targets for writing "metadata", meaning assets and timeseries name, description, and metadata. By default the extractor will create time series with just an external ID and nothing else. | | **metadata-targets.clean** | Configuration for writing metadata to CDF Clean. | | **metadata-targets.clean.assets** | Set to `true` to enable creating assets in CDF. | | **metadata-targets.clean.time-series** | Set to `true` to enable writing time series metadata. | | **max-upload-interval** | Maximum time to cache datapoints before they are uploaded to CDF. The syntax is described in [Timestamps and intervals](#timestamps). Defaults to `1s` | | **max-data-points-upload-queue-size** | Maximum number of cached datapoints before they are uploaded to CDF. Defaults to `1000000`. | | **cdf-retries** | Configuration for automatic retries on requests to CDF. | | **cdf-retries.timeout** | Timeout in milliseconds for each individual try. Defaults to `80000`. | | **cdf-retries.max-retries** | The maximum number of retries. Less than 0 retries forever. | | **cdf-retries.max-delay** | Maximum delay between each try in milliseconds. Base delay is calculated according to `125 * 2 ^ retry` milliseconds. If this is less than 0, there is no upper limit. Defaults to `5000`. | | **cdf-chunking** | Configuration for chunking on requests to CDF. Note that increasing these may cause requests to fail, due to limits in the API. Read the API documentation before making these higher than their current value. | | **cdf-chunking.time-series** | Maximum number of time series per get/create time series request. | | **cdf-chunking.assets** | Maximum number of assets per get/create assets request. | | **cdf-chunking.data-point-time-series** | Maximum number of time series per datapoint create request. | | **cdf-chunking.data-points** | Maximum number of datapoints per datapoint create request. | | **cdf-throttling** | Configuration for how requests to CDF should be throttled. | | **cdf-throttling.time-series** | Maximum number of parallel requests per time series operation. Defaults to `20`. | | **cdf-throttling.assets** | Maximum number of parallel requests per assets operation. Defaults to `20`. | | **cdf-throttling.data-points** | Maximum number of parallel requests per datapoints operation. Defaults to `10`. | | **sdk-logging** | Configuration for logging of requests from the SDK. | | **sdk-logging.disable** | Set this to `true` to disable logging of requests from the SDK, it's enabled by default. | | **sdk-logging.level** | Log level to log messages from the SDK at, defaults to `debug`. | | **sdk-logging.format** | Format of the log message. Defaults to `CDF ({Message}): {HttpMethod} {Url} - {Elapsed} ms` | | **nan-replacement** | Replacement for `NaN` values when writing to CDF. Defaults to none, meaning these are just removed. | | **extraction-pipeline.external-id** | Configuration for associating this extractor with an extraction pipeline. Used for monitoring and remote configuration. | | **certificates** | Configuration for special handling of SSL certificates. This shouldn't be considered a permanent solution to certificate problems. | | **certificates.accept-all** | Accept all remote SSL certificates even if verification fails. This introduces a risk of man-in-the-middle attacks. | | **certificates.allow-list** | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates. | ## State Store Configuration for storing state in a local database or in CDF RAW. This is required if reading from an HDA server. | Parameter | Description | | - | - | | **location** | Path to database file, or name of raw database containing state store. | | **database** | Which type of database to use. Valid options are `LiteDb`, `Raw`, or `None`. The default value is `None`. | | **interval** | Interval between each push of local states to the state store. The syntax is described in [Timestamps and intervals](#timestamps). The default value is `1m`. | ## History Configuration for reading historical data from an HDA server. | Parameter | Description | | - | - | | **backfill** | Set this to `true` to enable backfill, meaning that the extractor will read backwards from the earliest known timestamp as well as forwards from the latest known timestamp on startup. This is only useful if there is enough data in the server that reading it all will take a very long time. | | **start-time** | The earliest timestamp history will be read from, in milliseconds since `01/01/1970`. Alternatively use syntax `N[timeunit]-ago` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is in the past, otherwise it will be in the future. | | **end-time** | The latest timestamp that history will be read from, in milliseconds since `01/01/1970`. Alternatively use syntax `N[timeunit]-ago` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is in the past, otherwise it will be in the future. | | **chunking** | Chunking for history reads. | | **chunking.chunk-size** | Maximum number of items per history read request. Defaults to `1000`. | | **chunking.parallelism** | Maximum number of parallel history read requests. Defaults to `10`. | | **chunking.max-per-minute** | Maximum number of history read requests per minute. | | **chunking.max-read-per-tag** | Maximum number of values returned per tag per request. Defaults to `1000`. | | **granularity** | Granularity to use when doing history read. Nodes with the last/earliest known timestamp within this range of each other will be read together. This shouldn't be smaller than the usual average update rate. Leave at 0 to always read a single node each time. The syntax is described in [Timestamps and intervals](#timestamps). Defaults to `15s`. | # Hardware and server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_classic/opc_classic_requirements Hardware and server requirements for the OPC Classic extractor. ## Hardware The hardware requirements of the OPC Classic extractor depend on the configuration and server size. ### Memory The extractor replicates the entire server in memory, and will buffer up to a few minutes of data points and events at a time. Running the extractor against a small server should take no more than 100 MB of memory, while running against a larger server (>100k tags) might take as much as 2-3 GB. If the server is relatively small, a VM with a single core and 1 GB of memory should be sufficient. ### CPU For most cases, a single CPU core is sufficient for the extractor. ### Software The extractor must run on a Windows machine with COM+ enabled. ### Storage The extractor writes log files to disk. This should usually not take up much space. # Set up the OPC Classic extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_classic/opc_classic_setup Install and configure the OPC Classic extractor on Windows with support for DCOM connections. The Cognite OPC Classic extractor is distributed as a Windows installer (.msi file). When you run the .msi file, a setup wizard guides you through the installation process. The installer creates a Windows service and installs the binary files needed to run the extractor. ## Before you start * Check the [requirements](/cdf/integration/guides/extraction/opc_classic/opc_classic_requirements) for the extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#opc-classic-extractor) in a Cognite Data Fusion (CDF) project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). * Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime. * [Download and run the installer](/cdf/integration/concepts/extraction#download-installation-files) for the extractor from CDF. Set Modify permission under **Properties** in the installation folder to avoid permission issues. * In the `config` folder, copy `config.minimal.yml` to `config.yml` and edit to fit your requirements. * Start the extractor by running the executable file from the folder above the `config` folder, `.\bin\OpcClassicExtractor.exe`, or by starting the Windows service. ## Connecting over DCOM The extractor supports connecting to servers over Distributed component object model (DCOM), but the setup for this is not trivial. This is only required if the extractor is on a different machine than the server. The setup is more straightforward if the server runs on the same machine as the extractor. You must follow these instructions for **both** the server **and** the client since DCOM is bidirectional. 1. Enter the firewall configuration under `Control Panel > System and Security > Windows Defender Firewall`. 2. Choose `Advanced settings`. 3. Go to `Inbound Rules` and add a `New Rule...`. Configure the first rule: 1. `Rule Type` should be `Custom`. 2. Under `Program`, select `All programs`. 3. Under `Protocol and Ports`, set `Protocol type` to `TCP`, set `Local port` to `RPC Endpoint Mapper`, set `Remote port` to `All Ports`. 4. Under `Scope`, set local IP addresses to `Any IP address`. We recommend setting remote IP addresses to the IP address of the extractor if it resides on a server, or the server's IP address if the extractor resides on a client. 5. Under `Action`, select `Allow the connection`. 6. Under `Profile`, select according to your setup. You typically set this to `Private`, but you may need to update the network profile used by the extractor. 7. Enter a name, for instance `DCOM Discovery`. Create another rule: 1. Set `Rule Type` to `Custom`. 2. Under `Program`, select `All programs`. 3. Under `Protocol and Ports`, set `Protocol type` to `TCP`, set `Local port` to `RPC Dynamic Ports`, set `Remote port` to `All Ports`. 4. Under `Scope`, set local IP addresses to `Any IP address`. We recommend setting remote IP addresses to the IP address of the extractor if this resides on a server, or the server's IP address if the extractor resides on a client. 5. Under `Action`, select `Allow the connection`. 6. Under `Profile`, select according to your setup. You typically set this to `Private`, but you may need to update the network profile used by the extractor. 7. Enter a name, for instance `DCOM Dynamic Ports`. This also needs to be done on both the server and the client, though the configuration is slightly different. 1. Open the Start menu > `Run` program. 2. Type in `DCOMCNFG`. This opens the configuration for DCOM. 3. You'll find the configuration under `Component Services > Computers > My Computer`. 4. Under `DCOM Config`, grant permissions to the user the extractor will run as: * Find the server in the list of DCOM applications. * Right-click and select `Properties`. * Set `Authentication Level` to `Connect`. * Under `Security`, select `Customize` and `Edit` for each permission. * Grant access to `Local Launch`, `Remote Launch`, `Local Activation`, `Remote Activation`, `Local Access`, `Remote Access`, and `Read`. It may be possible to restrict this further, depending on the server. 5. Make the same changes to `OpcEnum`. 6. Under `My Computer > Properties` in the same view: * Go to `Default Properties`. * `Enable Distributed COM on this computer`. * Set `Default Authentication Level` to `Connect`. * Set `Default Impersonation Level` to `Identity`. # About the Cognite OPC UA extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua Learn about the Cognite OPC UA extractor that reads time series, events, and asset information via the OPC UA protocol and streams data and events to time series in Cognite Data Fusion (CDF). **Deploy with the Cognite Toolkit:** The [Foundation deployment pack](/cdf/deploy/cdf_toolkit/references/packages/foundation) includes a `cdf_opcua_extractor` module with ready-made extraction pipeline configs — use it instead of configuring the extractor pipeline by hand. The OPC UA extractor supports the following: * **Security** using a username/password or automatically generated client/server certificates. * **Mapping a node hierarchy** from an arbitrary root node to CDF. * **Subscribing to data changes** in OPC UA and pushing these to destinations. * **Pushing to more than one destination type**, such as CDF and InfluxDB. You can configure only one InfluxDB database per extractor. * **Reading historical data** from OPC UA, starting from the earliest last point found in all push destinations. * **Using properties** in OPC UA as metadata in CDF. * **Numerical data types** in OPC UA. By default, these are basic types, but you can add additional types in the configuration. * **String values**: Non-array data types in OPC UA can be converted to strings and then pushed to CDF. * **State time series** for data modeling projects, mapping OPC UA enumeration and multi-state variables to a single plottable time series with numeric values and string labels when enabled in the configuration. * **Variables** in OPC UA with an array-type data type of fixed length can be mapped to different measurements in influx or time series in CDF. * **Setting up a list of events** during the configuration. The event properties are automatically mapped and used as metadata in CDF. * **Using different emitters** for events than the server node. * **Discovering new nodes** through auditing or by periodically re-browsing the node tree. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-32 OPC UA extractor configuration reference for version 2.32. This page documents OPC UA extractor **2.32**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`stream-records`](#cognite.stream-records) | object | BETA: Write events to data modeling stream records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `stream-records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling stream records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-33 OPC UA extractor configuration reference for version 2.33. This page documents OPC UA extractor **2.33**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-34 OPC UA extractor configuration reference for version 2.34. This page documents OPC UA extractor **2.34**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-35 OPC UA extractor configuration reference for version 2.35. This page documents OPC UA extractor **2.35**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-36 OPC UA extractor configuration reference for version 2.36. This page documents OPC UA extractor **2.36**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-37 OPC UA extractor configuration reference for version 2.37. This page documents OPC UA extractor **2.37**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-38 OPC UA extractor configuration reference for version 2.38. This page documents OPC UA extractor **2.38**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | | `metadata-as-json` | boolean | Controls whether metadata is written as JSON objects or as a dictionary. When `true`, metadata is serialized as JSON in the extractedData field. When `false`, metadata is stored as a dictionary of key-value pairs. cognite.metadata-targets.`clean.space` is required when using `metadata-as-json` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/2-39 OPC UA extractor configuration reference for version 2.39. This page documents OPC UA extractor **2.39**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. :::tip Tip You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ::: ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | | `metadata-as-json` | boolean | Controls whether metadata is written as JSON objects or as a dictionary. When `true`, metadata is serialized as JSON in the extractedData field. When `false`, metadata is stored as a dictionary of key-value pairs. cognite.metadata-targets.`clean.space` is required when using `metadata-as-json` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/3-0 OPC UA extractor configuration reference for version 3.0. This page documents OPC UA extractor **3.0**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, no metadata is written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, no metadata is written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | | `metadata-as-json` | boolean | Controls whether metadata is written as JSON objects or as a dictionary. When `true`, metadata is serialized as JSON in the extractedData field. When `false`, metadata is stored as a dictionary of key-value pairs. cognite.metadata-targets.`clean.space` is required when using `metadata-as-json` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata and `enums-as-strings` to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less means that the extractor will not write to the state store. It will still be read from on startup. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `30s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # Command-line arguments Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_cli Reference guide for all available command-line arguments when running the OPC UA extractor, including options for configuration, authentication, and tool mode. | Option | Description | | - | - | | tool | Launch the [configuration tool](/cdf/integration/guides/extraction/opc_ua/opcua) instead of the extractor. Use the configuration tool to analyze the server and suggest a suitable set of configuration options. | | -e\|--endpoint-url \[host] | Override the **source.endpoint-url** configuration parameter and connect to the specified port. This generally has to be on the form `opc.tcp://…:port`. | | -u\|--user \[username] | Override the Source/Username option and use the specified username to connect to the OPC UA server. | | -p\|--password \[password] | Override the Source/Password option and use the specified password to connect to the OPC UA server. | | --secure | Allow the extractor to try to connect to a secure endpoint if possible. This corresponds to the Source/Secure configuration option. | | -f\|--config-file \[path] | Override the location of the YAML configuration file. This can be a relative or absolute path. | | --auto-accept | Auto-accept connections from unrecognized servers. You can set this the first time the extractor or the configuration tool connects to the server and then remove it to ensure that the client only accepts connections from a server with the same certificate as the first time. | | -d\|--config-dir \[dir] | Override the path to the configuration files. This is overwritten by the **config-file** option for YAML configuration files but will still apply to the OPC UA XML configuration file. You can also set this using the `OPCUA_CONFIG_DIR` environment variable. | | -o\|--config-target \[path] | Set the output path for the configuration tool. This should be a YAML file, and it will be overwritten. If this is the same as the configuration file, then (mostly) the results will be written back. We recommend that you use a different file. | | -n\|--no-config | Don't load any YAML configuration file. The OPC UA XML configuration file is still needed, but as long as the necessary information is provided using command-line options, the extractor or the configuration tool can be run in a simple configuration without any configuration file. | | -l\|--log-level \[level] | Override the **logger**/**console**/**level** option. One of `verbose`, `debug`, `information`, `warning`, `error`. | | -w\|--working-dir \[dir] | Set the working directory of the extractor. Relative paths are relative to this value. Default is the current directory for standalone or one level above executable if starting as a service. | | -s\|--service | Required when starting the extractor as a service. This option won't start the extractor as a service on its own, but makes the necessary modifications when it's called from the system. | | -x\|--exit | Exit the extractor on failure. Overrides the Source/ExitOnFailure configuration option. | | --dry-run | Run the extractor in dry run mode. In this mode, the extractor will not push anything to any destination. | | --version | Show the current extractor version. | | -?\|--h--help | Show the available command line options. | # Hardware requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_hw_requirements Memory, CPU, and storage requirements for running the OPC UA extractor based on server size and configuration. ## Memory At some points, the extractor replicates the entire node hierarchy and buffers up to a few minutes of data points and events at a time. Running the extractor against a small server should take no more than 100 MB of memory, while running against a larger server (> 1 million nodes) may take as much as 3–4 GB. If the server is relatively small, a VM with a single core and 1 GB of memory should be sufficient. ## CPU In most cases, the bottleneck is the OPC UA server and the network, not the extractor. The OPC UA client operates over a single TCP connection, and the extractor will generally not have to do more than some simple transformations of the data arriving over that connection. Heavy use of complicated regex transformations can cause issues, but those only apply on startup. Regular operation of the extractor doesn't require much processing power. ## Storage By default, the extractor doesn't write to disk, apart from these exceptions: * If `failure-buffer` is configured to buffer data points to disk, the extractor may write an arbitrarily large amount of data. The data points are buffered directly, and if the connection to Cognite Data Fusion (CDF) is down for a few hours, with a few hundred data points per second, this may grow fairly large. If the intent is to use the buffer, and the server doesn't support history, we recommend giving the extractor a decent amount of permanent storage. The data points are removed once the connection to CDF is re-established. * The extractor is configured to write logs to disk. These will generally not be very large unless verbose logging is enabled. We don't recommend verbose logging for production use. # Metrics Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics Complete reference of available Prometheus metrics from the OPC UA extractor, including source system metrics, extractor metrics, CDF metrics, InfluxDB metrics, MQTT metrics, history metrics, and native Prometheus metrics. For information on the different types of metrics, see the official [Prometheus documentation](https://prometheus.io). In general, **Counter** starts at zero and increases during a run of the extractor, **Gauge** may change freely, **Summary** calculates a running average of observations and **Histogram** contains counts of different measurement sizes, distributed in buckets. ## Source system | Name | Type | Purpose | | - | - | - | | **opcua\_version** | Counter | Value always `0`, help string contains version of extractor. | | **opcua\_connects** | Counter | Counts the number of times the extractor has connected to the OPC UA server. | | **opcua\_connected** | Gauge | 1\`\`if the extractor is connected to the server,`0` otherwise. Can be used to track extractor connectivity, together with Prometheus heartbeat. | | **opcua\_attribute\_requests** | Counter | Number of `Read` service calls made to the OPC UA server. | | **opcua\_subscriptions** | Gauge | Number of data variables currently subscribed to. | | **opcua\_history\_reads** | Counter | Number of `HistoryRead` calls made to the OPC UA server. | | **opcua\_browse\_operations** | Counter | Number of `Browse` service calls made to the OPC UA server. | | **opcua\_attribute\_request\_failures** | Counter | Number of times calls to the `Read` service has failed. | | **opcua\_history\_read\_failures** | Counter | Number of times calls to the `HistoryRead` service has failed. | | **opcua\_browse\_failures** | Counter | Number of times calls to the `Browse` service has failed. | | **opcua\_bad\_datapoints** | Counter | Number of data points with a `Bad` status code received from the server. | | **opcua\_bad\_events** | Counter | Number of events skipped due to bad fields. | ## Extractor | Name | Type | Purpose | | - | - | - | | **opcua\_start\_time** | Gauge | Start time for the extractor, in milliseconds since 1/1/1970 | | **opcua\_extractor\_starting** | Counter | `1` if the extractor is in the process of starting, `0` otherwise. | | **opcua\_tracked\_assets** | Counter | Number of OPC UA nodes that have been mapped to destination assets. | | **opcua\_tracked\_timeseries** | Counter | Number of OPC UA nodes that have been mapped to destination time series. | | **opcua\_num\_pushes** | Counter | Increments by 1 for each push of data points and events to destinations. | | **opcua\_streaming\_delay\_datapoints** | Summary | Estimated time it takes for data points to go from the source system to the extractor. Relies on clocks being synchronized between extractor machine and source machines.

    With this, average time to CDF can be calculated as (`1 / rate(opcua_num_pushes)) / 2 + opcua_streaming_delay_data` | | **opcua\_streaming\_delay\_events** | Summary | Similarly for events, time from source system to extractor. Note that this uses the `Time` property, which may be set by the server in some cases, or be generated in the past. | | **opcua\_array\_points\_missed** | Counter | Number of points missed due to nodes reporting too large arrays. | | **opcua*node*\[node name]** | Gauge(s) | The value of nodes in the OPC UA server hierarchy, if configured. | ## CDF | Name | Type | Purpose | | - | - | - | | **opcua\_datapoints\_pushed\_cdf** | Counter | Number of data points pushed to CDF. | | **opcua\_datapoint\_pushes\_cdf** | Counter | Number of times data points have been pushed to CDF. | | **opcua\_datapoint\_push\_failures\_cdf** | Counter | Number of failed pushes of data points to CDF. | | **opcua\_events\_pushed\_cdf** | Counter | Number of events pushed to CDF. | | **opcua\_event\_pushes\_cdf** | Counter | Number of times events have been pushed to CDF. | | **opcua\_event\_push\_failures\_cdf** | Counter | Number of failed pushes of events to CDF. | | **opcua\_node\_ensure\_failures\_cdf** | Counter | Number of times trying to create nodes in Clean or RAW have failed completely. | | **opcua\_skipped\_events\_cdf** | Counter | Number of events skipped due to issues or because they already exist. | | **opcua\_missing\_timeseries** | Gauge | Number of time series missing from CDF. | | **opcua\_mismatched\_timeseries** | Gauge | Number of time series with wrong datatype in CDF relative to OPC UA. | | **extractor\_utils\_cdf\_asset\_requests** | Summary | Number and duration of asset requests to CDF. | | **extractor\_utils\_cdf\_timeseries\_requests** | Summary | Number and duration of time series requests to CDF. | | **extractor\_utils\_cdf\_datapoint\_requests** | Summary | Number and duration of data point requests to CDF. | | **extractor\_utils\_cdf\_event\_requests** | Summary | Number and duration of event requests to CDF. | | **extractor\_utils\_cdf\_raw\_requests** | Summary | Number and duration of raw requests to CDF. | | **extractor\_utils\_cdf\_assets\_skipped** | Counter | Number of assets skipped due to errors. | | **extractor\_utils\_cdf\_timeseries\_skipped** | Counter | Number of time series skipped due to errors. | | **extractor\_utils\_cdf\_events\_skipped** | Counter | Number of events skipped due to errors. | | **extractor\_utils\_cdf\_login\_requests** | Summary | Number and duration of requests to /login/status | | **extractor\_utils\_cdf\_token\_requests** | Summary | Number and duration of requests to /token/inspect | | **extractor\_utils\_cdf\_datapoints** | Counter | Number of data points pushed to CDF by the utils. | | **extractor\_utils\_cdf\_invalid\_data\_points** | Counter | Number of data points skipped due to bad timestamp. | | **cognite\_sdk\_fetch\_inc** | Counter | Number of `POST/GET/...` actions performed | | **cognite\_sdk\_fetch\_error\_inc** | Counter | Number of errors on actions. | | **cognite\_sdk\_fetch\_retry\_inc** | Counter | Number of retries on actions. | | **cognite\_sdk\_decode\_error\_inc** | Counter | Decoding data errors. | | **cogntie\_sdk\_fetch\_latency\_update** | Gauge | Latency on actions performed. | ## Influx | Name | Type | Purpose | | - | - | - | | **opcua\_datapoint\_pushes\_influx** | Counter | Number of times data points have been pushed to influxdb. | | **opcua\_datapoints\_pushed\_influx** | Counter | Number of data points pushed to influxdb. | | **opcua\_datapoint\_push\_failures\_influx** | Counter | Number of failed pushes of data points to influxdb. | | **opcua\_skipped\_datapoints\_influx** | Counter | Number of data points that have been skipped while pushing to influxdb. | | **opcua\_event\_pushes\_influx** | Counter | Number of times events have been pushed to influxdb. | | **opcua\_events\_pushed\_influx** | Counter | Number of events pushed to influxdb. | | **opcua\_event\_push\_failures\_influx** | Counter | Number of failed pushes of events to influxdb. | | **opcua\_events\_skipped\_influx** | Counter | Number of events that have been skipped while pushing to influxdb. | ## MQTT | Name | Type | Purpose | | - | - | - | | **opcua\_created\_assets\_mqtt** | Counter | Number of assets sent by the MQTT pusher. | | **opcua\_created\_timeseries\_mqtt** | Counter | Number of time series sent by the MQTT pusher. | | **opcua\_datapoints\_pushed\_mqtt** | Counter | Number of data points sent by the MQTT pusher. | | **opcua\_datapoint\_pushes\_mqtt** | Counter | Number of times data points have been sent by the MQTT pusher. | | **opcua\_skipped\_datapoints\_mqtt** | Counter | Number of data points skipped by the MQTT pusher. | | **opcua\_events\_pushed\_mqtt** | Counter | Number of events sent by the MQTT pusher. | | **opcua\_event\_pushes\_mqtt** | Counter | Number of times events have been sent by the MQTT pusher,. | | **opcua\_skipped\_events\_mqtt** | Counter | Number of events skipped by the MQTT pusher due to invalid data. | | **opcua\_created\_relationships\_mqtt** | Counter | Number of relationships sent by the MQTT pusher. | ## History, State Store and Failure Buffer | Name | Type | Purpose | | - | - | - | | **opcua\_frontfill\_data\_points** | Counter | Number of data points retrieved through frontfill. | | **opcua\_frontfill\_data\_count** | Counter | Number of frontfill data points requests to OPC UA server. | | **opcua\_backfill\_data\_points** | Counter | Number of data points retrieved through backfill. | | **opcua\_backfill\_data\_count** | Counter | Number of backfill data points requests to OPC UA server. | | **opcua\_frontfill\_events** | Counter | Number of events retrieved through frontfill. | | **opcua\_frontfill\_events\_count** | Counter | Number of frontfill events requests to OPC UA server. | | **opcua\_backfill\_events** | Counter | Number of events retrieved through backfill. | | **opcua\_backfill\_events\_count** | Counter | Number of backfill events requests to OPC UA server. | | **opcua\_buffer\_num\_points** | Gauge | Number of data points stored in buffer. | | **opcua\_buffer\_num\_events** | Gauge | Number of events stored in buffer. | | **extractor\_utils\_restore\_states** | Counter | Number of states retrieved from state store. | | **extractor\_utils\_restore\_count** | Counter | Number of times states have been retrieved from state store. | | **extractor\_utils\_store\_states** | Counter | Number of states stored in state store. | | **extractor\_utils\_store\_count** | Counter | Number of times states have been written to state store. | ## Native Prometheus | Name | Type | Purpose | | - | - | - | | **dotnet\_total\_memory\_bytes** | Gauge | Number of bytes of allocated memory | | **dotnet\_collection\_count\_total** | Counter | Number of GC calls, grouped by generation. | | **process\_cpu\_seconds\_total** | Counter | Total number of CPU seconds used by the process. | | **process\_num\_threads** | Gauge | Current threads in use by the process. | | **process\_open\_handles** | Gauge | Current open handles. | | **process\_open\_fds** | Gauge | Current open file descriptors. | | **process\_private\_memory\_bytes** | Gauge | Current private memory bytes. | | **process\_resident\_memory\_bytes** | Gauge | Current resident memory bytes. | | **process\_virtual\_memory\_bytes** | Gauge | Current virtual memory. | | **process\_working\_set\_bytes** | Gauge | Current working set. | | **process\_start\_time\_seconds** | Gauge | Start time in unix time seconds. | | **push\_time\_seconds** | Gauge | Time in unix time seconds data was last pushed to the Prometheus pushgateway. | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_server_requirements Server requirements for running the OPC UA extractor with different features, including session management, variables, history, arrays, events, and auditing. Depending on the features used, the OPC UA extractor places certain requirements on the server. If these aren't fulfilled, the feature won't work, and the extractor may crash. ## Base These services are required for all servers the OPC UA extractor connects to: * **Session management** - The server must support connections from clients and be able to manage at least one session. * **Browse and BrowseNext** - These services are used for mapping the node hierarchy and must be supported. If the server doesn't support **BrowseNext**, it must return all child nodes on a single request. It's possible to get around this limitation by adjusting the `browse-chunk` and `browse-nodes-chunk` configuration parameters. * **Read** - The server must support the **Read** service to read attributes for nodes. ## Variables Any server that has data variables to read from must support these: * **Create Subscription** - The client must be able to create new subscriptions. * **Create Monitored Item** - The client must be able to create new monitored items on an existing subscription. * **Publish** - The server must handle publish requests from the client in a good way. * **Notifications on data changes** - The server must provide clean notifications on data changes to the client. ## History To read from a server with **History** set to `true`, the server must support the following: * **HistoryReadRaw** - Data is only read using the **ReadRaw** mode history read. The server must support this. * **ContinuationPoints** - The server must return a `continuationPoint` if a single request is insufficient to fetch all data points for a given node. * **Historizing** - Any variable that supports history must have its **Historizing** attribute set to `true`. ## Backfill To backfill, the server must support the following: * **History read ordering** - If start/endtime is reversed in a `historyRead` call, the data must be returned in reverse order. If not, backfill may not work properly. ## Arrays To read from a server with a non-zero `MaxArraySize`, the following must be true: * **ArrayDimensions** - Must be of length `1`, and be set for all variables that should be read. * **ValueRank** - Must be set to `0` or higher, as `-1` indicates a scalar. ## Events To read events from the server, it must support the following: * **Create Subscription** - The client must be able to create new subscriptions. * **Create Monitored Item** - The client must be able to create new `monitoredItems` on an existing subscription. * **Publish** - The server must handle publish requests from the client in a good way. * **Notifications on events** - The server must provide clean notifications on events to the client, on the form specified in the specifications. * **Filtering of events** - The server must support event filters, with at a minimum the `InList`, `Equals`, and `And` operators. The extractor filters clientside, but the server must tolerate a filter option, at the least, and respect the `SelectClauses` to deliver only requested fields. * **SourceNode** - The server must set the `SourceNode` property on all events that the extractor will map to destinations. * **EventNotifier** - The server should correctly set the `EventNotifier` property. If this is not done, event emitters must be manually added. ## History In addition to the base event properties, to read history, the server must support: * **HistoryReadEvents** - Reading events from history. * **ContinuationPoints** - The server must return a `continuationPoint` if a single request is insufficient to fetch all events for a given emitter. * **Filtering in history** - Same as the general case, but for history. Note that in both cases, a server can ignore filters, though the `SelectClause` must be respected. ## Auditing In addition to the base event properties, to support node discovery using auditing, the server must support: * **AuditingEvents** - Of at least two types, `AuditAddNodesEventType`, and `AuditAddReferencesEventType`. These must be emitted from the server node and provided to the client. # Set up the OPC UA extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_setup Instructions for installing and configuring the Cognite OPC UA extractor as a Windows executable, Windows service, Linux service, or Docker container. The steps involve using the configuration tool, which makes many calls to the OPC UA server. If you're not sure the server can handle the load, you can configure the extractor manually. To create and monitor the extractor as an **integration** (public preview), see [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration). After the connection succeeds, use [Monitor integrations](/cdf/integration/guides/interfaces/monitor_unified_integrations) to review **tasks**. ## Scope the OPC UA server before you configure Before you deploy configuration files to production pipelines, follow these principles: * **Prefer a connectivity gateway when you can**: Determine whether the extractor connects through an OPC UA aggregation server or other industrial connectivity gateway, or directly to embedded hardware such as a PLC or DCS controller. Gateways generally provide caching and tolerate higher concurrency. * **Protect direct controller connections**: Direct PLC or DCS connections have limited computing capacity and can crash if you overload them with large parallel queries. * **Scope root nodes**: Avoid browsing the entire root namespace wildcard. Configure specific root nodes (`RootNode`) to map only the asset subtrees, equipment hierarchies, and telemetry tags you need. * **Defer chunk sizes to server specifications**: To maintain compliance and prevent data truncation or timeout crashes, honor the server's intrinsic chunk limits. Keep `limit-to-server-config: true` enabled in your configuration and avoid overriding individual chunk properties unless server defaults are demonstrably invalid. * **Throttle historical backfill**: Historical data ingestion operates concurrently with real-time streaming. Configure throttling limits (such as `history.throttling.max-per-minute` and `history.throttling.max-parallelism`) to ensure backfill operations do not degrade real-time telemetry streaming or saturate the source server. ## Set up the extractor Check the [hardware](/cdf/integration/guides/extraction/opc_ua/opc_ua_hw_requirements) and [server](/cdf/integration/guides/extraction/opc_ua/opc_ua_server_requirements) requirements for the extractor. [Download the installer/installation files](/cdf/integration/concepts/extraction#download-installation-files-and-documentation) for the extractor from Cognite Data Fusion (CDF). Install the extractor based on your operating system. Run the installer. Extract the contents of the installation zip file to the desired location. If writing to the location requires administrator rights, the extractor must be run as admin or granted access in another way. In general, we recommend that you don't run the extractor as an admin. Run the configuration tool. This runs the configuration tool without a configuration file and connects to the host using the specified username and password (optional). The tool outputs the resulting configuration to `config/config.yml`. For more details, see [The configuration tool steps](#the-configuration-tool-steps). ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} OpcuaExtractor tool -n -e [host] -u [username] -p [password] --auto-accept -o config/config.yml ``` The extractor by default expects to find the configuration file in a local folder called `config`. If you are running the extractor from the command line with the default installation, it should be run from the install folder: `bin\OpcuaExtractor.exe ...` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} opcua-extractor tool -n -e [host] -u [username] -p [password] --auto-accept -o config/config.yml ``` The extractor by default expects to find the configuration file in a local folder called `config`. When setting up an extractor against a new server, you should run the configuration tool to detect potential server issues and ensure a successful start to the configuration. If the configuration tool fails, the server may be non-compliant, have insufficient computing power, or scale poorly. You may still be able to run the extractor, but this will require manual configuration and debugging. If you choose not to run the configuration tool, Cognite won't offer any support. Edit `config/config.yml` to fit your requirements. The most important elements are setting a suitable ID prefix and adding pushers, metrics, and logging. This might also involve disabling some features that the configuration tool discovered or adding ones that it didn't. Optionally, edit the `opc.ua.net.extractor.Config.xml` file used for configuring the OPC UA SDK. For most users, the default settings are suitable, but in some cases, you may need to change memory limits or similar. Start the extractor with the default settings, loading the configuration from `config/config.yml`. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} OpcuaExtractor ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} opcua-extractor ``` ## The configuration tool steps This section lists each action the configuration tool takes when testing the server. In general, the configuration tool is designed to build on the existing configuration and can update an existing configuration file. Usually, it's ok to let it run with no configuration. Attempts to treat the given `endpointUrl` as a discovery server and look for endpoints there. If more secure options are available, it might suggest connecting to these. This should be cheap and quick. Issues have been found with some servers that return too few results with certain settings for the number of nodes browsed at a time or the number of desired results. This section browses part of the node tree several times, trying different parts of the hierarchy and different configurations. It may also try to browse the server hierarchy if the main node tree is too small. If the number of nodes is very large, the smallest chunks are skipped. The root of browsing is, by default, the objects folder, but you can set this using `RootNode`. Looks through the data type hierarchy, and identifies any likely candidates for custom numeric data types, that you want to map to double in destination systems. Tries to find a suitable limit for the `Read` service, reading attributes for as many nodes as possible at a time. Typically, it starts at 1000, then 100 and 10 at a time. Using the retrieved attributes, looks for arrays and strings that you want to push to CDF, then find suitable settings for the relevant configuration options. Tries to subscribe to as many nodes as possible at a time, again starting at 1000 and going down. It will also listen and see if it receives any data on any of the subscriptions. Tries to read history from nodes with **Historizing** set. If no such nodes exist, this step is skipped. The extractor tries to read the first and last parts of history and estimate the number of points in each. If the number is sufficiently large, the configuration tool will suggest enabling `Backfill`. This consists of several steps: 1. Look through the event type hierarchy and try to find custom events. 2. Look for `GeneratesEvent` references between nodes. 3. Look at the `EventNotifier` property to supply information about which nodes are allowed to emit events. If both `GeneratesEvent` and `EventNotifiers` are used on the server, the intersection of the two are used to find emitters. 4. Check if the auditing property on the server is set. If it's set, then enable auditing. Parses the namespace map to suggest a suitable shortening of each namespace to create compact external IDs. The configuration tool outputs the final configuration file with all detected settings. ## Run as a Docker container Use a Docker container to run the OPC UA extractor on macOS or Linux. There is no native macOS binary. The image is available on Docker Hub as `cognite/opcua-extractor-net`. Consider sharing a folder if a deployment uses more than one file, for example, logs and local state files. The Docker container comes with three volumes: * `config` contains the configuration file. This is optional if the extractor is set up with remote configuration. * `logs` contains the log files, if enabled. * `certificates` contains the application and server certificates. This volume *must* be configured. If you mount a host folder on `/config`, that folder must contain `opc.ua.net.extractor.Config.xml`. An empty folder hides the default file in the image, and the extractor fails with `Missing opc.ua.net.extractor.Config.xml in config folder /config`. Run the extractor with this command. Replace `` with the extractor version you want to run. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} docker run -v "$(pwd)/config:/config" -v "$(pwd)/certificates:/certificates" cognite/opcua-extractor-net: ``` # Extracting state time series with the OPC UA extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_state_timeseries Enable opt-in state time series ingestion from OPC UA multi-state and enumeration variables in data modeling projects. The Cognite OPC UA extractor can ingest both the numeric value and string label from OPC UA state variables into a single plottable [state time series](/dev/concepts/resource_types/state_timeseries) in data modeling projects. Charts and analysis tools can plot the numeric state code while labels stay readable in tooltips and legends. This feature is in [public preview](/cdf/product_feature_status#public-preview) and may change. ## Before you start * Use **Cognite OPC UA extractor 3.0.0** or later. * Set up the extractor through [Integrations](/cdf/integration/guides/interfaces/about_unified_integrations) (public preview). State time series ingestion is available for OPC UA extractors registered as on-premises integrations. * Configure the extractor to write to a **data modeling project**, not asset-centric data only. * Assign **data model instances** capabilities (`datamodelinstances:read` and `datamodelinstances:write`) for the target space. See [Capabilities for the OPC UA extractor](/cdf/access/guides/capabilities#opc-ua-extractor). * Complete [Set up the OPC UA extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup) with a working connection to your OPC UA server. ## What the extractor creates When [`extraction.data-types.states.enabled`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states) is `true`, the extractor: 1. Creates **state sets** (`CogniteStateSet`) for supported OPC UA variables. 2. Creates **state time series** with `type: "state"` and links each series to its state set. 3. Writes **state data points** with a numeric state code and a human-readable label on each change. The feature is **opt-in**; the config defaults to `false`, so existing extractor behavior is unchanged until you enable it. ## Supported OPC UA state variables The extractor chooses multi-state, enumeration, or two-state discrete handling automatically during browse. * **Multi-state**: States are defined on the variable (one tag, one state list). OPC UA types include `MultiStateDiscreteType` and `MultiStateValueDiscreteType`. * **Enumeration**: States are defined on a shared type (many tags, one state list). Set [`extraction.data-types.auto-identify-types`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.auto-identify-types) to `true`. The extractor reads state definitions from `DataTypeDefinition` when available (OPC UA 1.05) and uses `EnumStrings` or `EnumValues` only when `DataTypeDefinition` does not define them. * **Two-state discrete**: Boolean variables with two states defined on the server (for example active and inactive). OPC UA type is `TwoStateDiscreteType`. ## Existing time series Enabling state time series ingestion does not convert or delete existing time series. The extractor does not delete time series or data points. If a variable already has a string or numeric time series in CDF, the extractor skips creating a state time series for that variable. Search extractor logs for `skipping creation of state timeseries`. If you previously ingested the variable as a string time series and want state time series instead, delete the existing time series and reingest after you enable [`extraction.data-types.states.enabled`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states). This avoids duplicate series for the same variable. Whether you migrate is optional and depends on your downstream consumers. ## Enable state time series extraction Edit `config.yml` and set `extraction.data-types.states.enabled` to `true` under the [`data-types`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types) section. Set `extraction.data-types.auto-identify-types` to `true` when you ingest enumeration variables. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} extraction: data-types: auto-identify-types: true states: enabled: true ``` See the [`states` configuration reference](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states) for parameter descriptions. Restart the extractor as described in [Start the extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup#start-the-extractor) so it rebrowses the OPC UA node tree and applies the new settings. Use the [data modeling API](/dev/concepts/resource_types/state_timeseries) to confirm that ingestion succeeded: 1. **`CogniteStateSet` instances** exist in CDF: one per multi-state variable, or one shared set per enumeration type. 2. Affected time series have `type: "state"` and link to a state set. 3. Recent data points include a **numeric state code** and a **human-readable label**. ## Limitations * **Data modeling projects only** — State time series extraction does not apply when writing only to asset-centric data. * **Opt-in** — Disabled by default; enable `extraction.data-types.states.enabled` explicitly. * **100-state limit (platform)** — CDF state sets support at most 100 states. If an OPC UA variable defines more than 100 states at browse, the extractor does not create a state set or state time series and ingests the variable as a **regular time series** instead. If a state set later grows past 100 states, the extractor logs an error and stops updating that time series. * **State set updates** — Changes to state definitions on the OPC UA server are picked up on **rebrowse** only, not during live subscription. Until CDF reflects the updated state set, state codes outside the current CDF state set are written with status **`BadOutOfRange`** and have no value. * **Errors in logs** — Limit violations and invalid state values are logged by the extractor; they are not surfaced in the CDF UI. See [Troubleshoot the OPC UA extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_troubleshooting). # Troubleshoot the OPC UA extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_troubleshooting Solutions and instructions for troubleshooting common problems when setting up the OPC UA extractor to push data to Cognite Data Fusion (CDF). We recommend accessing the server through the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool to verify endpoints and inspect node structures. This assumes that you have first run the configuration tool as described in [the configuration settings](/cdf/integration/guides/extraction/opc_ua/opcua). ## Connection and certificate issues If the extractor is unable to connect to the OPC UA server or maintain a secure session, try these solutions. Do not leave `source.auto-accept: true` or `source.ignore-certificate-issues: true` enabled in production. These flags disable certificate validation and expose telemetry and credentials to interception. Use them only for short diagnostic tests while you set up certificate trust. See [Configure SSL certificates and proxy settings for extractors](/cdf/integration/guides/extraction/ssl_certificates_and_proxies). If you get the error message `Unable to connect to discovery server`, make sure that you have specified the correct **endpoint-url** in [the configuration file](/cdf/integration/guides/extraction/opc_ua/opcua) and that it's possible to connect to the server from where the extractor is running. **Solution**: 1. Confirm **endpoint-url** in [the configuration file](/cdf/integration/guides/extraction/opc_ua/opcua). 2. From the extractor host, connect with [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) to verify network reachability. **Cause**: The username or password in the configuration file or environment variables does not match what the OPC UA server expects. **Solution**: Verify the credentials in the configuration file or environment variables, then restart the extractor. **Cause**: The OPC UA server does not trust the extractor's client certificate. OPC UA uses mutual server/client certificate validation, so the connection is rejected. **Solution**: 1. Move the extractor's client certificate into the OPC UA server's trusted certificate store. 2. For a short connectivity test during initial setup, you can temporarily set `source.auto-accept: true` in the configuration file, then set it back to `false`. See the warning in [Connection and certificate issues](#connection-and-certificate-issues). For comprehensive certificate and proxy guidance across runtime environments, see [Configure SSL certificates and proxy settings for extractors](/cdf/integration/guides/extraction/ssl_certificates_and_proxies). This typically occurs if the server can't handle a high load. Try to inspect what the extractor was doing as it crashed, then reduce `source/browse-nodes-chunk`, `source/browse-chunk`, `source/attributes-chunk`, or `source/subscription-chunk` in [the configuration file](/cdf/integration/guides/extraction/opc_ua/opcua). **Solution**: 1. Reduce chunk sizes in `config.yml`: reduce `source/browse-nodes-chunk`, `source/browse-chunk`, `source/attributes-chunk`, or `source/subscription-chunk`. 2. Increase `source/keep-alive-interval`. **Cause**: The server certificate is signed with SHA1. The extractor treats SHA1 certificates as insecure and rejects them by default. **Solution**: * Update the server certificate to use SHA-256. * As a temporary diagnostic workaround only, set `false` under `SecurityConfiguration` in `opc.ua.net.extractor.Config.xml`, or set `source.ignore-certificate-issues: true` in `config.yml`. See the warning in [Connection and certificate issues](#connection-and-certificate-issues). **Cause**: The server certificate key length is below the required security minimum. **Solution**: * Increase the key length of the server certificate. * Alternatively, adjust `MinimumCertificateKeySize` under `SecurityConfiguration` in `opc.ua.net.extractor.Config.xml`, or set `source.ignore-certificate-issues: true` as a temporary diagnostic workaround only. See the warning in [Connection and certificate issues](#connection-and-certificate-issues). **Cause**: The server certificate lacks the data-encipherment usage flag required by the OPC UA standard. **Solution**: Reissue the server certificate with standard data encipherment key usage flags, or set `source.ignore-certificate-issues: true` as a temporary diagnostic workaround only. See the warning in [Connection and certificate issues](#connection-and-certificate-issues). **Cause**: The secure channel was terminated because of a server-side session timeout, network interruption, certificate expiration, or server restart. **Solution**: 1. Check the OPC UA server logs for session disconnect reasons. 2. Increase `session-timeout` and `keep-alive-interval` in the extractor configuration file. *** ## Browse and hierarchy issues If the extractor encounters errors when traversing the server hierarchy or discovering nodes, try these solutions. **Cause**: Server-side resource constraints or concurrency limits fail when the extractor browses a large node tree. **Solution**: Reduce the browse load on the server: * Set `browse-chunk: 100` and `browse-nodes-chunk: 100` in the configuration. * Set `max-parallelism: 1` and `max-node-parallelism: 1` to serialize queries. **Cause**: The OPC UA server's response exceeds message encoding limits because the namespace is large or a single response returns too many child nodes. **Solution**: 1. Reduce `browse-chunk` and `browse-nodes-chunk` in `config.yml` (for example, down to `5` or `10`). 2. If using the .NET extractor, increase message buffer limits in `opc.ua.net.extractor.Config.xml`: * `MaxStringLength` * `MaxMessageSize` * `MaxByteStringLength` * `MaxArrayLength` **Cause**: The server invalidated browse continuation points after a nightly restart, a maintenance window, or a short session continuation timeout. **Solution**: 1. Reduce `browse-nodes-chunk` and `browse-chunk` to complete browse operations faster within single sessions. 2. Set `max-parallelism: 1`. 3. Set `ignore-continuation-points: true` in the configuration to re-initiate browse queries when continuation points become invalid. *** ## Bad or missing data If the extractor is unable to retrieve data, skips variables, or receives data in the wrong format from the OPC UA server, try these solutions. **Cause**: The server has no mappable variables, string or non-scalar types are filtered out, or root-node and filter settings omit the nodes. **Solution**: 1. In [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html), confirm that variables appear as green labels. 2. Try setting `extraction.data-types.allow-string-variables` to `true`. 3. If time series are still missing, check their dimensions. By default, the extractor requires variables to be scalar or fixed-size 1D arrays. Try setting `extraction.data-types.unknown-as-scalar` to `true` if variables have `ValueRank` equal to `Any` or `OneOrMoreDimensions`. 4. Verify that the variables are not excluded by filters or omitted root node configurations. **Cause**: The extractor infers data types from server definitions, and some variables are missing type metadata. If you write to a data modeling project and the variable is an OPC UA enumeration or multi-state type, enable `extraction.data-types.states.enabled` to ingest both the numeric code and string label into a [state time series](/dev/concepts/resource_types/state_timeseries). See [Extracting state time series with the OPC UA extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_state_timeseries). **Solution**: 1. Set `extraction.data-types.auto-identify-types` to `true`. 2. Set `extraction.data-types.null-as-numeric` to `true` to treat variables with null `DataType` as numeric (only do this if you are certain none of these should be strings). If this is the case, either verify that the variables are not excluded by filters or omitted root node configurations, or contact the server administrator. Unknown data types aren't supported by the extractor, except with this workaround. **Cause**: The extractor maps OPC UA variables to time series and properties to metadata. The server represents some metadata as variables. **Solution**: Use `extraction.transformations` in the configuration file to explicitly select and remap specific variable nodes to metadata. **Cause**: The extractor requires scalar values or fixed-size arrays by default, so it skips variables with undefined array dimensions. **Solution**: Set `extraction.data-types.unknown-as-scalar: true` to treat dynamic or unspecified arrays as scalar values. You can also disable `estimate-array-sizes` to reduce startup evaluation overhead. **Cause**: Historical backfill queries stale NodeIDs. This can happen if the raw node buffer skips node re-discovery after structural changes on the source server. **Solution**: 1. Disable `raw-node-buffer` in the configuration file to force fresh node discovery. 2. Clean up stale historical state in the CDF staging area if needed. *** ## State time series If the extractor does not create state time series, fails to push state sets, omits string labels, or does not pick up state definition changes, try these solutions. **Cause**: State time series ingestion is disabled, the extractor is not set up through Integrations, the extractor is not writing to a data modeling project, the service principal lacks write access, or the OPC UA variable is not a supported type. **Solution**: 1. Verify that `extraction.data-types.states.enabled` is set to `true` in [the configuration file](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states). 2. Confirm that the extractor writes to a data modeling project. 3. Confirm that the service principal has [`datamodelinstances:write`](/cdf/access/guides/capabilities#opc-ua-extractor) for the target space. 4. Confirm that the OPC UA variable matches a [supported variable type](/cdf/integration/guides/extraction/opc_ua/opc_ua_state_timeseries#supported-opc-ua-state-variables). The extractor chooses multi-state, enumeration, or two-state discrete handling automatically during browse; there is no separate configuration for that choice. 5. Confirm that the extractor is set up through [Integrations](/cdf/integration/guides/interfaces/about_unified_integrations) (public preview). 6. For enumeration variables, set `extraction.data-types.auto-identify-types` to `true`. 7. Restart the extractor as described in [Start the extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup#start-the-extractor) so it rebrowses the node tree. 8. If multi-state variables are still missing after restart, set `extraction.data-types.states.allow-non-standard-types` to `true`. See the [`states` configuration reference](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states). **Cause**: The extractor could not write `CogniteStateSet` instances to CDF. **Solution**: Search extractor logs for `Failed to push state sets in CDF`. See [OPC UA extractor metrics](/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics) for log configuration. **Cause**: The variable already has a numeric or string time series in CDF, often from earlier ingestion with [`enums-as-strings`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.enums-as-strings). Enabling [`extraction.data-types.states.enabled`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction.data-types.states) does not convert or delete that series. The extractor does not delete time series or data points. **Solution**: 1. Search extractor logs for `skipping creation of state timeseries`. The existing time series is unchanged. 2. If you previously ingested the variable as a string time series and want state time series instead, delete the existing time series and reingest after you enable state time series ingestion. This avoids duplicate series for the same variable. Whether you migrate is optional. **Cause**: The OPC UA variable defines more than 100 states at browse, which exceeds the CDF platform limit for state sets. **Solution**: The extractor ingests the variable as a regular time series and does not create a `CogniteStateSet` or state time series. Search extractor logs for `more than the 100 a CDF state set allows`. See [OPC UA extractor metrics](/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics) for log configuration. **Cause**: The OPC UA variable initially had 100 or fewer states, but the state set in CDF later grew past the 100-state platform limit. **Solution**: The extractor logs an error and stops updating the time series. Search extractor logs for `has accumulated` and `more than the 100 a CDF state set allows`. Reconcile the state definitions on the OPC UA server or plan ingestion without a state time series for that variable. See [OPC UA extractor metrics](/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics) for log configuration. **Cause**: The extractor received a state code or label that is not in the CDF state set. It writes a data point with status **`BadOutOfRange`** and no value. **Solution**: 1. Search extractor logs for `Writing a bad state data point with no value`, `is not one of the`, and `states its state set defines`. See [OPC UA extractor metrics](/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics) for log configuration. 2. Verify that OPC UA state definitions on the server match the state set in CDF. 3. If the definitions changed recently, [restart the extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup#start-the-extractor) or set [`auto-rebrowse-period`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction) to refresh state sets. **Cause**: The extractor does not detect live changes to state definitions on the OPC UA server. Updated state sets appear in CDF only after a rebrowse. Until then, state codes outside the old state set are written with status **`BadOutOfRange`** and have no value. **Solution**: Restart the extractor as described in [Start the extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup#start-the-extractor) to trigger a rebrowse. Alternatively, set [`auto-rebrowse-period`](/cdf/integration/guides/extraction/opc_ua/opcua#extraction) in the configuration file. **Cause**: `FiniteStateMachineType` is not a supported OPC UA state variable type. **Solution**: Ingest **multi-state** variables (one state list per tag) or **enumeration** variables (one shared state list per type) instead. See [Supported OPC UA state variables](/cdf/integration/guides/extraction/opc_ua/opc_ua_state_timeseries#supported-opc-ua-state-variables). # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opcua Complete reference for configuring the Cognite OPC UA extractor version 2.40. This page documents OPC UA extractor **2.40** (current release). For older releases, see **Other versions** in the sidebar. To configure the OPC UA extractor, you must edit the configuration file. The file is in [YAML](https://yaml.org/) format, and the sample configuration file contains all valid options with default values. You can leave many fields empty to let the extractor use the default values. The configuration file separates the settings by component, and you can remove an entire component to disable it or use the default values. ## Sample configuration files In the extractor installation folder, the `/config` subfolder contains sample complete and minimal configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 1 of the configuration schema. Not that it is *not* recommended to use the `config.example.yml` as a basis for configuration files. This file contains *all* configuration options, which is both hard to read, and may cause issues. It is intended as a reference showing how each option is configured, *not* as a basis. Use `config.minimal.yml` instead. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Minimal YAML configuration file ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 1 source: # The URL of the OPC-UA server to connect to endpoint-url: 'opc.tcp://localhost:4840' cognite: # The project to connect to in the API, uses the environment variable COGNITE_PROJECT. project: '${COGNITE_PROJECT}' # Cognite authentication # This is for Microsoft as IdP. To use a different provider, # set implementation: Basic, and use token-url instead of tenant. # See the example config for the full list of options. idp-authentication: # Directory tenant tenant: ${COGNITE_TENANT_ID} # Application Id client-id: ${COGNITE_CLIENT_ID} # Client secret secret: ${COGNITE_CLIENT_SECRET} # List of resource scopes, ex: # scopes: # - scopeA # - scopeB scopes: - ${COGNITE_SCOPE} extraction: # Global prefix for externalId in destinations. Should be unique to prevent name conflicts. id-prefix: 'gp:' # Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. # Saves space compared to using the full URL. Using the ns index is not safe as the order can change on the server. # It is recommended to set this before extracting the node hierarchy. # For example: # NamespaceMap: # "urn:cognite:net:server": cns # "urn:freeopcua:python:server": fps # "http://examples.freeopcua.github.io": efg ``` ## ProtoNodeId You can provide an OPC UA `NodeId` in several places in the configuration file, these are YAML objects with the following structure: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node: node-id: i=123 namespace-uri: opc.tcp://test.test ``` To find the node IDs we recommend using the [UAExpert](https://www.unified-automation.com/products/development-tools/uaexpert.html) tool. Locate the node you need the ID of in the hierarchy, the find the node ID on the right side under **Attribute** > **NodeId**. Find the **Namespace Uri** by matching the **NamespaceIndex** on the right to the dropdown on the left, in the node hierarchy view. The default value is **No highlight**. ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For history start and end times you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the extractor to read only recent history. History start and end times can also be set to a specific date on RFC 3339 form, like `2024-01-02T09:27:15Z`, or on a shorter form like `2021-07-23+01:00`. A timezone specifier is necessary, `Z` means UTC. ## Using values from Azure Key Vault The OPC UA extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Configure the OPC UA extractor OPC-UA extractor configuration | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | `dry-run` | boolean | Set this to `true` to prevent the extractor from writing anything to CDF. This is useful for debugging the extractor configuration. | | [`source`](#source) | object | | | [`logger`](#logger) | object | Configure logging to console or file | | [`metrics`](#metrics) | object | Configure logging to console or file | | [`cognite`](#cognite) | object | Configuration for pushing data to Cognite Data Fusion (CDF) | | [`mqtt`](#mqtt) | object | Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | | [`influx`](#influx) | object | Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | | [`extraction`](#extraction) | object | Configuration for general extraction options, such as data types, mapping, and filters. | | [`events`](#events) | object | Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | | [`failure-buffer`](#failure-buffer) | object | If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | | [`history`](#history) | object | Configuration for reading historical datapoints and events from the server | | [`state-storage`](#state-storage) | object | A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events.

    Enabling this is highly recommended, and will be required in a future version of the extractor. | | [`subscriptions`](#subscriptions) | object | A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions:
    \* DataChangeListener - handles data point subscriptions.
    \* EventListener - handles event subscriptions.
    \* AuditListener - which handles audit events.
    \* NodeMetrics - which handles subscriptions for use as metrics.

    Each of these can contain a number of MonitoredItems. | | [`pub-sub`](#pub-sub) | object | Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | | [`high-availability`](#high-availability) | object | Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | ##
    `source` Global parameter. | Parameter | Type | Description | | - | - | - | | `endpoint-url` | string | The URL of the OPC UA server to connect to.

    In practice, this is the URL of the discovery server, where multiple levels of severity may be provided. The OPC UA extractor attempts to use the highest security possible based on the configuration.

    **Example:**
    `opc.tcp://some-host:1883` | | [`alt-endpoint-urls`](#source.alt-endpoint-urls) | list | List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/).

    We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. | | [`endpoint-details`](#source.endpoint-details) | object | Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | | [`redundancy`](#source.redundancy) | object | Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | | `reverse-connect-url` | string | The local URL used for reverse connect, which means that the server is responsible for initiating connections, not the extractor. This lets the server be behind a firewall, forbidding incoming connections. You must also specify an [`endpoint-url`](#source.endpoint-url), to indicate to the extractor where it should accept connections from. | | `auto-accept` | boolean | Set to `true` to automatically accept server certificates.

    If this is disabled, received server certificates will be placed in the rejected certificates folder (by default `application_dir/certificates/pki/rejected`), and you can manually move them to te accepted certificates folder (`application_dir/certificates/pki/accepted`). Setting this to `true` makes the extractor move certificates automatically.

    A simple solution would be to set this to `true` for the first connection, then change it to `false`.

    Warning: This should be disabled if the extractor is running on an untrusted network. Default value is `True`. | | `username` | string | OPC UA server username, leave empty to disable username/password authentication. | | `password` | string | OPC UA server password. | | [`x509-certificate`](#source.x509-certificate) | object | Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | | `secure` | boolean | Set this to `true` to make the extractor try to connect to an endpoint with security above `None`. If this is enabled, the extractor will try to pick the *most secure* endpoint, meaning the endpoint with the longest of the most modern cipher types. | | `ignore-certificate-issues` | boolean | Ignore all suppressible certificate errors on the server certificate. You can use this setting if you receive errors such as *Certificate use not allowed*.

    CAUTION: This is potentially a security risk. Bad certificates can open the extractor to man-in-the-middle attacks or similar. If the server is secured in other ways (it is running locally, over a secure VPN, or similar), it is most likely fairly harmless.

    Some errors are not suppressible and must be remedied on the server. Note that enabling this is always a workaround for the server violating the OPC UA standard in some way. | | `publishing-interval` | integer | Sets the interval (in milliseconds) between publish requests to the server, which is when the extractor asks the server for updates to any active subscriptions.

    This limits the maximum frequency of points pushed to CDF, but not the maximum frequency of points on the server. In most cases, this can be set to the same as [`extraction.data-push-delay`](#extraction.data-push-delay). If set to `0` the server chooses the interval to be as low as it supports. Be aware that some servers set this lower limit very low, which may create considerable load on the server. Default value is `500`. | | `force-restart` | boolean | If `true`, the extractor will not attempt to reconnect using the OPC UA reconnect protocol if connection is lost, but instead always create a new connection. Only enable this if reconnect is causing issues with the server. Even if this is disabled, the extractor will generally fall back on regular reconnects if the server produces unexpected errors on reconnect. | | `exit-on-failure` | boolean | If `true`, the OPC UA extractor will be restarted completely on reconnect. Enable this if the server is expected to change dramatically while running, and the extractor cannot keep using state from previous runs. | | `keep-alive-interval` | integer | Specifies the interval in milliseconds between each keep-alive request to the server. The connection times out if a keep-alive request fails twice (`2 * interval + 100ms`). This typically happens if the server is down, or if it is hanging on a heavy operation and doesn't manage to respond to keep alive requests. Set this higher if keep alives often time out without the server being down. Default value is `5000`. | | `restart-on-reconnect` | boolean | If `true`, the OPC UA extractor will be restarted after reconnecting to the server. This may not be required if the server is the server is expected to not change much, and that it handles reconnects well. | | [`node-set-source`](#source.node-set-source) | object | Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | | `alt-source-background-browse` | boolean | If `true`, browses the OPC UA node hierarchy in the background when obtaining nodes from an alternative source, such as CDF Raw or NodeSet2 files. | | `limit-to-server-config` | boolean | Uses the `Server/ServerCapabilities` node in the OPC UA server to limit chunk sizes. Set this to `false` only if you know the server reports incorrect limits and you want to set them higher. If the real server limits are exceeded, the extractor will typically crash. Default value is `True`. | | `browse-nodes-chunk` | integer | Sets the maximum number of nodes per call to the Browse service. Large numbers are likely to exceed the server's tolerance. Lower numbers greatly increase startup time. Default value is `1000`. | | `browse-chunk` | integer | Sets the maximum requested results per node for each call to the Browse service. The server may decide to return fewer. Setting this lower increases startup times. Setting it to 0 leaves the decision up to the server. Default value is `1000`. | | `attributes-chunk` | integer | Specifies the maximum number of attributes to fetch per call to the Read service. If the server fails with `TooManyOperations` during attribute read, it may help to lower this value. This should be set as high as possible for large servers. Default value is `10000`. | | `subscription-chunk` | integer | Sets the maximum number of new `MonitoredItems` to create per operation. If the server fails with `TooManyOperations` when creating monitored items, try lowering this value. Default value is `1000`. | | [`browse-throttling`](#source.browse-throttling) | object | Settings for throttling browse operations. | | `certificate-expiry` | integer | Specifies the default application certificate expiration time in months. You can also replace the certificate manually by modifying the `opc.ua.net.extractor.Config.xml` configuration file. Note that the default values was changed as of version 2.5.3. Default value is `60`. | | [`retries`](#source.retries) | object | Specify the retry policy for requests to the OPC UA server. | ###
    `alt-endpoint-urls` Part of [`source`](#source) configuration. List of alternative endpoint URLs the extractor can attempt when connecting to the server. Use this for non-transparent redundancy. See the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). We recommend setting `force-restart` to `true`. Otherwise, the extractor will reconnect to the same server each time. Each element of this list should be a string. ### `endpoint-details` Part of [`source`](#source) configuration. Details used to override default endpoint behavior. This is used to make the client connect directly to an OPC UA endpoint, for example if the server is behind NAT (Network Address Translation), circumventing server discovery. | Parameter | Type | Description | | - | - | - | | `override-endpoint-url` | string | Endpoint URL to override URLs returned from discovery. This can be used if the server is behind NAT, or similar URL rewrites. | ### `redundancy` Part of [`source`](#source) configuration. Additional configuration options related to redundant servers. The OPC UA extractor supports *Cold* redundancy, as described in the [OPC UA standard part 4, section 6.6.2](https://reference.opcfoundation.org/v104/Core/docs/Part4/6.6.2/). | Parameter | Type | Description | | - | - | - | | `service-level-threshold` | integer | Servers above this threshold are considered live. If the server drops below this level, the extractor will switch, provided `monitor-service-level` is set to `true`. Default value is `200`. | | `reconnect-interval` | string | If using redundancy, the extractor will attempt to find a better server with this interval if service level is below threshold. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `10m`. | | `monitor-service-level` | boolean | If `true`, the extractor will subscribe to changes in `ServiceLevel` and attempt to change server once it drops below [`service-level-threshold`](#source.redundancy.service-level-threshold).

    This also prevents the extractor from updating states while service level is below the threshold, letting servers inform the extractor that they are not receiving data from all sources, and history should not be trusted. Once the service level goes back above the threshold, history will be read to fill any gaps. | ###
    `x509-certificate` Part of [`source`](#source) configuration. Specifies the configuration for using a signed x509 certificate to connect to the server. Note that this is highly server specific. The extractor uses the provided certificate to sign requests sent to the server. The server must have a mechanism to validate this signature. Typically the certificate must be provided by the server. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to local x509-certificate | | `password` | string | Password for local x509-certificate file | | `store` | either `None`, `Local` or `User` | Local certificate store to use. One of `None` (to use a file), `Local` (for LocalMachine) or `User` for the `User` store. Default value is `None`. | | `cert-name` | string | Name of certificate in store. Required to use `store`

    **Example:**
    `CN=MyCertificate` | ###
    `node-set-source` Part of [`source`](#source) configuration. Read from NodeSet2 files instead of browsing the OPC UA node hierarchy. This is useful for certain smaller servers, where the full node hierarchy is known before-hand. In general, it can be used to lower the load on the server. | Parameter | Type | Description | | - | - | - | | [`node-sets`](#source.node-set-source.node-sets) | list | **Required.** List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files.

    Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml`

    **Example:**
    `[{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}]` | | `instance` | boolean | If `true`, the instance hierarchy is not obtained from the server, but instead read from the NodeSet2 files. | | `types` | boolean | If `true`, event types, reference types, object types, and variable types are obtained from NodeSet2 files instead of the server. | ####
    `node-sets` Part of [`node-set-source`](#source.node-set-source) configuration. List of nodesets to read. Specified by URL, file name, or both. If no name is specified, the last segment of the URL is used as file name. File name is where downloaded files are saved, and where the extractor looks for existing files. Note that typically, you will need to define all schemas your server schema depends on. All servers should depend on the base OPC UA schema, so you should always include `https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml` Each element of this list should be a configuration specifying a node set file. | Parameter | Type | Description | | - | - | - | | `file-name` | string | Path to nodeset file. This is either the place where the downloaded file is saved, or a previously downloaded file. | | `url` | string | URL of publicly available nodeset file. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [{'file-name': 'Server.NodeSet2.xml'}, {'url': 'https://files.opcfoundation.org/schemas/UA/1.04/Opc.Ua.NodeSet2.xml'}] ``` ### `browse-throttling` Part of [`source`](#source) configuration. Settings for throttling browse operations. | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ### `retries` Part of [`source`](#source) configuration. Specify the retry policy for requests to the OPC UA server. | Parameter | Type | Description | | - | - | - | | `timeout` | either string or integer | Global timeout. After this much time has passed, new retries will not be created. Set this to zero for no timeout. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `max-tries` | integer | Maximum number of attempts. 1 means that only the initial attempt will be made, 0 or less retries forever. Default value is `5`. | | `max-delay` | either string or integer | Maximum delay between attempts, incremented using exponential backoff. Set this to 0 for no upper limit. Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `0s`. | | `initial-delay` | either string or integer | Initial delay used for exponential backoff. Time between each retry is calculated as `min(max-delay, initial-delay * 2 ^ retry)`, where 0 is treated as infinite for `max-delay`. The maximum delay is about 10 minutes (13 retries). Syntax is `N[timeUnit]` where `timeUnit` is `d`, `h`, `m`, `s` or `ms`. Default value is `500ms`. | | [`retry-status-codes`](#source.retries.retry-status-codes) | list | List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. | #### `retry-status-codes` Part of [`retries`](#source.retries) configuration. List of additional OPC-UA status codes to retry on. In additional to defaults. Should be integer values from [http://www.opcfoundation.org/UA/schemas/StatusCode.csv](http://www.opcfoundation.org/UA/schemas/StatusCode.csv), or symbolic names as shown in the same .csv file. Each element of this list should be a string. ## `logger` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | | `ua-trace-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Capture OPC UA tracing at this level or above. | | `ua-session-tracing` | boolean | Log data sent to and received from the OPC-UA server.

    WARNING: This produces an enormous amount of logs, only use this when running against a small number of nodes, producing a limited number of datapoints, and make sure it is not turned on in production. | ###
    `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configure logging to console or file | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for a prometheus scrape server. | | [`push-gateways`](#metrics.push-gateways) | list | A list of push gateway destinations to push metrics to | | [`nodes`](#metrics.nodes) | object | Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for a prometheus scrape server. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for the server.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to host the prometheus scrape server on | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of push gateway destinations to push metrics to | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://localhost:9091` | | `job` | string | **Required.** Name of the job | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ###
    `nodes` Part of [`metrics`](#metrics) configuration. Configuration to treat OPC-UA nodes as metrics. Values will be mapped to `opcua_nodes_[NODE-DISPLAY-NAME]` in prometheus | Parameter | Type | Description | | - | - | - | | `server-metrics` | boolean | Map a few relevant static diagnostics contained in the `Server/ServerDiagnosticsSummary` node to prometheus metrics. | | [`other-metrics`](#metrics.nodes.other-metrics) | list | List of additional nodes to read as metrics. | #### `other-metrics` Part of [`nodes`](#metrics.nodes) configuration. List of additional nodes to read as metrics. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ## `cognite` Global parameter. Configuration for pushing data to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`data-set`](#cognite.data-set) | object | Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | | `read-extracted-ranges` | boolean | Specifies whether to read start/end-points for datapoints on startup, where possible. It is generally recommended to use the [`state-storage`](#state-storage) instead of this. Default value is `True`. | | [`metadata-targets`](#cognite.metadata-targets) | object | Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | | [`metadata-mapping`](#cognite.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | | [`raw-node-buffer`](#cognite.raw-node-buffer) | object | Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`.

    This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all.

    If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | | `delete-relationships` | boolean | If this is set to `true`, relationships deleted from the source will be hard-deleted in CDF. Relationships do not have metadata, so soft-deleting them is not possible. | | [`records`](#cognite.records) | object | BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `data-set` Part of [`cognite`](#cognite) configuration. Data set used for new time series, assets, events, and relationships. Existing objects will not be updated | Parameter | Type | Description | | - | - | - | | `id` | integer | Data set internal ID | | `external-id` | string | Data set external ID | ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configure targets for node metadata. This configures which resources other than time series datapoints are created. By default, if this is left out, data is written to assets and time series metadata. Note that this behavior is deprecated, in the future leaving this section out will result in no metadata being written at all. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Write metadata to the CDF staging area (Raw). | | [`clean`](#cognite.metadata-targets.clean) | object | Write metadata to CDF clean, assets, time series, and relationships. | | [`data-models`](#cognite.metadata-targets.data-models) | object | ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support!

    This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`.

    Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | ####
    `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to the CDF staging area (Raw). | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF Raw database to write to. | | `assets-table` | string | | | `timeseries-table` | string | Name of the Raw table to write time series metadata to. | | `relationships-table` | string | Name of the Raw table to write relationships metadata to. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Write metadata to CDF clean, assets, time series, and relationships. | Parameter | Type | Description | | - | - | - | | `assets` | boolean | Set to `true` to enable creating CDF assets from OPC UA nodes. | | `timeseries` | boolean | Set to `true` to enable adding metadata to CDF time series based on OPC UA properties. | | `relationships` | boolean | Set to `true` to enable creating relationships from OPC UA references. Requires [`extraction.relationships`](#extraction.relationships) to be enabled. | | `space` | string | The Data Modeling `space` the extractor will create a Core Data Model compliant Time Series in. The space must have scoped access to the `CogniteExtractorTimeSeries` view. When defined, the extractor will write metadata to the Core Data Model CogniteExtractorTimeSeries view, backed by the CogniteTimeSeries container found in the Data Modeling space specified as an external ID. The Time Series data will also be avaialble from the CDF TimeSeries API using the 'instanceId' - `space.externalId` - of the Data Modeling Time Series instance. Note that at this time, the OPC UA extractor only supports the CogniteTimeSeries Core Data Model type

    This capability is Generally Available. | | `source` | string | External ID of the `source` that created core data model time series will be tied to. If this is not specified, it defaults to either `OPC_UA:[source.endpoint-url]` or `OPC_UA_NODESET:[source.node-set-source.nodesets[LAST].[file-name or url]` | | `metadata-as-json` | boolean | Controls whether metadata is written as JSON objects or as a dictionary. When `true`, metadata is serialized as JSON in the extractedData field. When `false`, metadata is stored as a dictionary of key-value pairs. cognite.metadata-targets.`clean.space` is required when using `metadata-as-json` | ####
    `data-models` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. ALPHA: Write metadata to ANY CDF Data Models. Please see the [`clean.space`](#clean.space) section of this configuration description for simplified Core Data Model Time Series support! This will create CDF data models based on the OPC UA type hierarchy, then populate them with data from the OPC UA node hierarchy. Note that this *requires* [`extraction.relationships.enabled`](#extraction.relationships) and [`extraction.relationships.hierarchical`](#extraction.relationships) to be set to `true`, and there must be exactly one root node with ID `i=84`. Note that this feature is in *alpha* there may be changes that require you to delete the data model from CDF, and breaking changes to the configuration schema. These changes will not be considered breaking changes to the extractor. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set this to `true` to enable writing to CDF Data Models. | | `model-space` | string | **Required.** Set the space to create data models in. The space will be created if it does not exist. | | `instance-space` | string | **Required.** Set the space instances will be created in. The space will be created if it does not exist. May be the same as `model-space`. | | `model-version` | string | **Required.** Version used for created data model and all created views. | | `types-to-map` | either `Referenced`, `Custom` or `All` | Configure which types to map to Data Models.

    `Referenced` means that only types that are referenced by instance nodes will be created.
    `Custom` means that all types not in the base namespace will be created.
    `All` means that all types will be created.

    Note: Setting this to `All` is rarely useful, and may produce impractically large models. Default value is `Custom`. | | `skip-simple-types` | boolean | Do not create views without their own connections or properties. Simplifies the model greatly, but reduces the number of distinct types in your model. | | `ignore-mandatory` | boolean | Let mandatory options be nullable. Many servers do not obey `Mandatory` requirements in their own models, which breaks when they are ingested into CDF, where nullable constraints are enforced. | | [`connection-target-map`](#cognite.metadata-targets.data-models.connection-target-map) | object | Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`.

    **Example:**
    `{'EquipmentType.EquipmentClass': 'ObjectType'}` | | `enable-deletes` | boolean | | #####
    `connection-target-map` Part of [`data-models`](#cognite.metadata-targets.data-models) configuration. Target connections on the form `"Type.Property": "Target"`. This is useful for certain schemas. This overrides the expected type of specific CDF Connections, letting you override incorrect schemas. For example, the published nodeset file for ISA-95 incorrectly states that the `EquipmentClass` reference for `EquipmentType` is an `Object`, while it should be an `ObjectClass`. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} EquipmentType.EquipmentClass: ObjectType ``` | Parameter | Type | Description | | - | - | - | | Any string matching `[A-z0-9-_.]+` | either `ObjectType`, `Object`, `VariableType`, `Variable`, `ReferenceType`, `DataType`, `View` or `Method` | NodeClass to override connection with. | ### `metadata-mapping` Part of [`cognite`](#cognite) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#cognite.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#cognite.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#cognite.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ### `raw-node-buffer` Part of [`cognite`](#cognite) configuration. Read from CDF instead of OPC-UA when starting, to speed up start on slow servers. Requires [`extraction.data-types.expand-node-ids`](#extraction.data-types) and [`extraction.data-types.append-internal-values`](#extraction.data-types) to be set to `true`. This should generaly be enabled along with [`metadata-targets.raw`](#cognite.metadata-targets.raw) or with no metadata targets at all. If `browse-on-empty` is set to `true` and `metadata-targets.raw` is configured with the same database and tables the extractor will read into raw on first run, then use raw as source for later runs. The Raw database can be deleted, and it will be re-created on extractor restart. | Parameter | Type | Description | | - | - | - | | `enable` | boolean | **Required.** Set to `true` to enable loading nodes from CDF Raw. | | `database` | string | **Required.** CDF RAW database to read from. | | `assets-table` | string | CDF RAW table to read assets from, used for events. This is not useful if there are no custom nodes generating events in the server. | | `timeseries-table` | string | CDF RAW table to read time series from. | | `browse-on-empty` | boolean | Run normal browse if nothing is found when reading from CDF, either because the tables are empty, or because they do not exist. Note that nodes may be present in the CDF RAW tables. Browse will run if no *valid* nodes are found, even if there are nodes present in RAW. | ### `records` Part of [`cognite`](#cognite) configuration. BETA: Write events to data modeling records. Containers corresponding to OPC-UA event types will be created as needed in the configured model-space. Note that if your server produces a lot of different event types, this can create a large number of containers. | Parameter | Type | Description | | - | - | - | | `log-space` | string | **Required.** Data modeling space to write the records to. The space will be created if it does not already exist. | | `model-space` | string | **Required.** Data modeling space to write containers to. The space will be created if it does not already exist. | | `stream` | string | **Required.** Stream to write the records to. The stream will be created if it does not already exist. | | `use-raw-node-id` | boolean | Use a raw string representation of node IDs, on the form `ns=NS;[i/s/o/g]=IDENTIFIER`. If this is not enabled, node IDs will be created as direct relations. | | `use-reversible-encoding` | boolean | Use OPC-UA reversible encoding for OPC-UA structures. This will make it possible to convert them back into OPC-UA types later, but it sacrifices human readability. | ## `mqtt` Global parameter. Push data to CDF one-way over [MQTT](https://mqtt.org/). This requires that the [MQTT-CDF Bridge](https://github.com/cognitedata/opcua-extractor-net/releases) application is running somewhere with access to CDF. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** The address of the MQTT broker.

    **Example:**
    `localhost` | | `port` | integer | **Required.** Port to connect to on the MQTT broker.

    **Example:**
    `1883` | | `username` | string | The MQTT broker username. Leave empty to connect without authentication. | | `password` | string | The MQTT broker password. Leave empty to connect without authentication. | | `use-tls` | boolean | Set this to `true` to enable Transport Level Security when communicating with the broker. | | `allow-untrusted-certificates` | boolean | Set this to `true` to allow untrusted SSL certificates when communicating with the broker. This is a security risk, we recommend using `custom-certificate-authority` instead. | | `custom-certificate-authority` | string | Path to certificate file for a certificate authority the broker SSL certificate will be verified against. | | `client-id` | string | MQTT client id. Should be unique for a given broker. Default value is `cognite-opcua-extractor`. | | `data-set-id` | integer | Data set to use for new assets, relationships, events, and time series. Existing objects will not be updated. | | `asset-topic` | string | Topic to publish assets on. Default value is `cognite/opcua/assets`. | | `ts-topic` | string | Topic to publish timeseries on. Default value is `cognite/opcua/timeseries`. | | `event-topic` | string | Topic to publish events on. Default value is `cognite/opcua/events`. | | `datapoint-topic` | string | Topic to publish datapoints on. Default value is `cognite/opcua/datapoints`. | | `raw-topic` | string | Topic to publish raw rows on. Default value is `cognite/opcua/raw`. | | `relationship-topic` | string | Topic to publish relationships on. Default value is `cognite/opcua/relationships`. | | `local-state` | string | Set to enable storing a list of created assets/timeseries to local litedb. Requires `state-storage.location` to be set. If this is left empty, metadata will have to be read each time the extractor restarts. | | `invalidate-before` | integer | Timestamp in ms since epoch to invalidate stored mqtt states. On extractor restart, assets/timeseries created before this will be re-created in CDF. They will not be deleted or updated. Requires the [`state-storage`](#state-storage) to be enabled. | | `skip-metadata` | boolean | Do not push any metadata at all. If this is true, plan timeseries without metadata will be created, like when using `raw-metadata`, and datapoints will be pushed. Nothing will be written to raw and no assets will be created. Events will be created, but without asset context | | [`raw-metadata`](#mqtt.raw-metadata) | object | Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected.
    Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | | [`metadata-mapping`](#mqtt.metadata-mapping) | object | Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that.

    **Example:**
    `{'timeseries': {'EngineeringUnits': 'unit', 'EURange': 'description'}, 'assets': {'Name': 'name'}}` | ###
    `raw-metadata` Part of [`mqtt`](#mqtt) configuration. Store assets/timeseries metadata and relationships in raw. Assets will not be created at all, timeseries will be created with just `externalId`, `isStep`, and `isString`. Both timeseries and assets will be persisted in their entirity to CDF Raw. Datapoints are not affected. Events will be created but without being contextualized to assets. The external ID of the source node is added to metadata if applicable | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** Raw database to write metadata to. | | `assets-table` | string | Raw table to use for assets. | | `timeseries-table` | string | Raw table to use for timeseries. | | `relationships-table` | string | Raw table to use for relationships. | ### `metadata-mapping` Part of [`mqtt`](#mqtt) configuration. Define mappings between properties in OPC UA and CDF attributes. For example, it is quite common for variables in OPC UA to have a property named `EngineeringUnits`, which ideally should be mapped to `unit` in CDF. This property lets you do that. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timeseries: EngineeringUnits: unit EURange: description assets: Name: name ``` | Parameter | Type | Description | | - | - | - | | [`assets`](#mqtt.metadata-mapping.assets) | object | Map metadata for assets. | | [`timeseries`](#mqtt.metadata-mapping.timeseries) | object | Map metadata for time series. | #### `assets` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for assets. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name` or `parentId` | Target asset attribute | #### `timeseries` Part of [`metadata-mapping`](#mqtt.metadata-mapping) configuration. Map metadata for time series. | Parameter | Type | Description | | - | - | - | | Any string | either `description`, `name`, `parentId` or `unit` | Target time series attribute | ## `influx` Global parameter. Configuration for pushing to an [InfluxDB](https://www.influxdata.com/) database. Data points and events will be pushed, but no context or metadata. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the host InfluxDB server | | `username` | string | The username for connecting to the InfluxDB database. | | `password` | string | The password for connecting to the InfluxDB database. | | `database` | string | **Required.** The database to connect to on the InfluxDB server. The database will not be created automatically. | | `point-chunk-size` | integer | Maximum number of points to send in each request to InfluxDB. Default value is `100000`. | | `read-extracted-ranges` | boolean | Whether to read start/end points on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | | `read-extracted-event-ranges` | boolean | Whether to read start/end points for events on startup, where possible. It is recommended that you use [`state-storage`](#state-storage) instead. | ## `extraction` Global parameter. Configuration for general extraction options, such as data types, mapping, and filters. ### External ID generation IDs used in OPC UA are special `nodeId` objects with an identifier and a namespace that must be converted to a string before they are written to CDF. A direct conversion, however, has several potential problems. * The namespace index is by default part of the node, but it may change between server restarts. Only the namespace itself is fixed. * The namespace table may be modified, in which case all old node IDs are invalidated. * Node IDs are not unique between different OPC UA servers. * Node identifiers can be duplicated accross namespaces. The solution is to create a node ID on the following form: `[id-prefix][namespace][identifierType]=[identifier as string]([optional array index])`. For example, the node with node ID `ns=1;i=123` with ID prefix `gp:` would be mapped to `gp:http://my.namespace.url:i=123`. You can optionally override this behavior for individual nodes by using `node-map`(#extraction.node-map). | Parameter | Type | Description | | - | - | - | | `id-prefix` | string | Global prefix for `externalId`s in destinations. Should be unique for each extractor deployment to prevent name conflicts. | | [`root-node`](#extraction.root-node) | object | Root node. Defaults to the Objects node. Default value is `{'node-id': 'i=86'}`. | | [`root-nodes`](#extraction.root-nodes) | list | List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided | | [`node-map`](#extraction.node-map) | object | Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. | | [`namespace-map`](#extraction.namespace-map) | object | Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | | `auto-rebrowse-period` | string | Time in minutes between each automatic re-browse of the node hierarchy. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), this option accepts cron expressions. Set this to `0` to disable automatic re-browsing of the server. Default value is `0m`. | | `enable-audit-discovery` | boolean | Enable this to make the extractor listen to `AuditAddNodes` and `AuditAddReferences` events from the server, and use that to identify when new nodes are added to the server. This is more efficient than browsing the node hierarchy, but does not work with data models and requires that the server supports auditing. | | `data-push-delay` | string | Time between each push to destinations. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `1s`. | | [`update`](#extraction.update) | object | Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | | [`data-types`](#extraction.data-types) | object | Configuration related to how data types and arrays should be handled by the OPC UA extractor. | | [`relationships`](#extraction.relationships) | object | Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]`

    Only relationships between mapped nodes are extracted. | | [`node-types`](#extraction.node-types) | object | Configuration for mapping OPC UA types to CDF in some way. | | `map-variable-children` | boolean | Set to `true` to make the extractor read children of variables and potentially map those to timeseries as well. | | [`transformations`](#extraction.transformations) | list | A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. | | [`deletes`](#extraction.deletes) | object | Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | | [`status-codes`](#extraction.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | | [`rebrowse-triggers`](#extraction.rebrowse-triggers) | object | Configuration for triggering rebrowse based on changes to specific nodes. | ###
    `root-node` Part of [`extraction`](#extraction) configuration. Root node. Defaults to the Objects node | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `root-nodes` Part of [`extraction`](#extraction) configuration. List of root nodes. The extractor will start exploring from these. Specifying nodes connected with hierarchical references can result in some strange behavior and should be avoided Each element of this list should be a root node. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `node-map` Part of [`extraction`](#extraction) configuration. Map from external IDs to OPC UA node IDs. This can, for example, be used to place the node hierarchy as a child of a specific asset in the asset hierarchy, or to manually rename certain nodes. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} myCustomExternalId: node-id: i=15 namespace-uri: urn:mynamespace ``` | Parameter | Type | Description | | - | - | - | | [Any string](#extraction.node-map.proto_node_id) | object | Target node ID for mapping external ID. Default value is `{'node-id': 'i=86'}`. | #### `proto_node_id` Part of [`node-map`](#extraction.node-map) configuration. Target node ID for mapping external ID. | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `namespace-map` Part of [`extraction`](#extraction) configuration. Map OPC-UA namespaces to prefixes in CDF. If not mapped, the full namespace URI is used. This saves space compared to using the full URI, and might make IDs more readable. | Parameter | Type | Description | | - | - | - | | Any string | string | Namespace URI to map. | ### `update` Part of [`extraction`](#extraction) configuration. Update data in destinations on re-browse or restart. Set `auto-rebrowse-period` to do this periodically. | Parameter | Type | Description | | - | - | - | | [`objects`](#extraction.update.objects) | object | Configuration for updating objects and object types. | | [`variables`](#extraction.update.variables) | object | Configuration for updating variables and variable types. | #### `objects` Part of [`update`](#extraction.update) configuration. Configuration for updating objects and object types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | #### `variables` Part of [`update`](#extraction.update) configuration. Configuration for updating variables and variable types. | Parameter | Type | Description | | - | - | - | | `description` | boolean | Set to `true` to update description | | `context` | boolean | Set to `true` to update context, i.e. the position of the node in the node hierarchy | | `metadata` | boolean | Set to `true` to update metadata, including fields like `unit`. | | `name` | boolean | Set to `true` to update name. | ### `data-types` Part of [`extraction`](#extraction) configuration. Configuration related to how data types and arrays should be handled by the OPC UA extractor. | Parameter | Type | Description | | - | - | - | | [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) | list | Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. | | [`ignore-data-types`](#extraction.data-types.ignore-data-types) | list | List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. | | `unknown-as-scalar` | boolean | Assume non-specific `ValueRanks` in OPC UA (`ScalarOrOneDimension` or `Any`) are scalar if they do not have an `ArrayDimensions` attribute. If such a variable produces an array, only the first element will be extracted. | | `max-array-size` | integer | Maximum length of arrays mapped to CDF, if this is set to 0, only scalar variables are mapped. Set this to `-1` to indicate that there is no upper limit.

    WARNING: If you set this to `-1`, and encounter a very large array, such as an image, the extractor might create hundreds of thousands of time series in CDF.

    In general, extracting arrays require variables to have an `ArrayDimensions` attribute with length equal to 1. See `estimate-array-sizes` for workarounds. | | `allow-string-variables` | boolean | Set to `true` to allow fetching string-typed variables. This means that all variables with a non-numeric type is converted to string. | | `auto-identify-types` | boolean | Map out the data type hierarchy before starting. This is useful if there are custom numeric types, or any enum types are used. This must be enabled for enum metadata, `enums-as-strings`, and [`states`](#extraction.data-types.states) to work. If this is false, any custom numeric types have to be added manually. | | `enums-as-strings` | boolean | If this is `false` and `auto-identify-types` is `true`, or there are manually added enums in `custom-numeric-types`, enums will be mapped to numeric time series, and labels are added as metadata fields.

    If this is `true`, labels are not mapped to metadata, and enums will be mapped to string time series with values equal to mapped label values.

    For data modeling projects, prefer [`states`](#extraction.data-types.states) when ingesting OPC UA enumeration or multi-state variables. | | [`states`](#extraction.data-types.states) | object | Configure ingestion of OPC UA state variables as [state time series](/dev/concepts/resource_types/state_timeseries) in data modeling projects. | | `data-type-metadata` | boolean | Add a metadata property `dataType`, which contains the node ID of the OPC-UA data type. | | `null-as-numeric` | boolean | Set this to `true` to treat `null` node IDs as numeric instead of ignoring them. Note that `null` node IDs violate the OPC UA standard. | | `expand-node-ids` | boolean | Add attributes such as `NodeId`, `ParentNodeId`, and `TypeDefinitionId` to nodes written to CDF Raw as full node IDs with reversible encoding. | | `append-internal-values` | boolean | Add attributes generally used internally like `AccessLevel`, `Historizing`, `ArrayDimensions`, and `ValueRank` to data pushed to CDF Raw. | | `estimate-array-sizes` | boolean | If `max-array-size` is set, this looks for the `MaxArraySize` property on each node with `one-dimensional` ValueRank. If it is not found, the extractor tries to read the value of the node, and look at the current array length.

    `ArrayDimensions` is still the preferred way to identify array sizes, this is not guaranteed to generate reasonable or useful values. | ####
    `states` Requires Cognite OPC UA extractor **3.0.0** or later. Available when the extractor is set up through [Integrations](/cdf/integration/guides/interfaces/about_unified_integrations) (public preview). Part of [`data-types`](#extraction.data-types) configuration. Ingest OPC UA enumeration, multi-state, and two-state discrete variables as [state time series](/dev/concepts/resource_types/state_timeseries) in data modeling projects. When [`enabled`](#extraction.data-types.states) is `true`, the extractor: 1. Creates **state sets** (`CogniteStateSet`) for supported OPC UA variables. 2. Creates **state time series** with `type: "state"` and links each series to its state set. 3. Writes **state data points** with a numeric state code and a human-readable label on each change. The extractor detects whether each variable is a multi-state, enumeration, or two-state discrete type during browse. You do not choose between them in configuration. See [Extracting state time series with the OPC UA extractor](/cdf/integration/guides/extraction/opc_ua/opc_ua_state_timeseries). | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Opt in to state time series ingestion. Default is `false`; existing extractor behavior is unchanged until you enable it. | | `allow-non-standard-types` | boolean | Set to `true` only when multi-state variables are not ingested and your OPC UA server uses non-standard Multi-State types. Default is `false`. See [troubleshooting when state time series are not created](/cdf/integration/guides/extraction/opc_ua/opc_ua_troubleshooting#state-time-series-are-not-created). | Type-level enumeration requires [`auto-identify-types`](#extraction.data-types.auto-identify-types) to be `true`. For enumeration types, the extractor prefers state definitions from `DataTypeDefinition` (OPC UA 1.05). It reads `EnumStrings` or `EnumValues` only when `DataTypeDefinition` does not define the states. #### `custom-numeric-types` Part of [`data-types`](#extraction.data-types) configuration. Set a list of manually defined numeric OPC UA data types. This can be used to make custom types be treated as numbers. Conversion from the actual returned values is done using the `C#` `Convert` functionality. If no valid conversion exists, this will fail. Each element of this list should be a definition of an OPC UA data type. | Parameter | Type | Description | | - | - | - | | [`node-id`](#extraction.data-types.custom-numeric-types.node-id) | object | Node ID of the data type. Default value is `{'node-id': 'i=86'}`. | | `is-step` | boolean | Set to `true` if is-step should be set on timeseries in CDF | | `enum` | boolean | Set to `true` if this is data type is an enumeration | ##### `node-id` Part of [`custom-numeric-types`](#extraction.data-types.custom-numeric-types) configuration. Node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | #### `ignore-data-types` Part of [`data-types`](#extraction.data-types) configuration. List of node IDs of data types that should not be extracted to CDF. Time series with one of these data types are not extracted. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `relationships` Part of [`extraction`](#extraction) configuration. Map OPC UA references to relationships in CDF, or edges in CDF data models. Generated relationships will have external ID on the form `[prefix][reference type];[source][target]` Only relationships between mapped nodes are extracted. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable mapping OPC-UA references to relationships in CDF | | `hierarchical` | boolean | Set to `true` to enable also mapping the hierarchical references over. These are normally mapped to `assetId`/`parentId` relations in CDF, but when doing that the type of the reference is lost. Requires `enabled` to be `true`. | | `inverse-hierarchical` | boolean | Set to `true` to create inverse relationships for each fo the hierarchical references. For efficiency these are not read, but inferred from forward references. Does nothing if `hierarchical` is `false`. | ### `node-types` Part of [`extraction`](#extraction) configuration. Configuration for mapping OPC UA types to CDF in some way. | Parameter | Type | Description | | - | - | - | | `metadata` | boolean | Add the `TypeDefinition` as a metadata field to all objects and variables. | | `as-nodes` | boolean | Allow reading object- and variable types as normal nodes and map them to assets or data model nodes in CDF. They will need to be in the mapped hierarchy, so you will need to add a root node that they are descended from. | ### `transformations` Part of [`extraction`](#extraction) configuration. A list of transformations to be applied to the source nodes before pushing. Transformations are applied sequentially, so it can help performance to put `Ignore` filters first, and `TimeSeries` filters can undo `Property` transformations. Each element of this list should be an a single transformation. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} filter: name: Cognite$ nodeClass: Object type: Include ``` | Parameter | Type | Description | | - | - | - | | `type` | either `Ignore`, `DropSubscriptions`, `Property`, `TimeSeries`, `AsEvents` or `Include` | Transformation type. `Ignore` ignores the node and all descendants, `DropSubscriptions` prevents the extractor from subscribing to this node, `Property` converts time series and their descendants into metadata, `TimeSeries` converts metadata into time series, `DropSubscriptions` prevents the extractor from subscribing to events or data points on matching nodes, `AsEvents` turns time series data points into events, `Include` inverts an ignore filter. If any `Include` filter is present, all nodes are ignored by default. | | [`filter`](#extraction.transformations.filter) | object | Filter to match. All non-null filters must match each node for the transformation to be applied. | #### `filter` Part of [`transformations`](#extraction.transformations) configuration. Filter to match. All non-null filters must match each node for the transformation to be applied. | Parameter | Type | Description | | - | - | - | | [`name`](#extraction.transformations.filter.name) | configuration for either File, Values or Regex | Regex or list of values matching node `DisplayName`. | | [`description`](#extraction.transformations.filter.description) | configuration for either File, Values or Regex | Regex or list of values matching node `Description`. | | [`id`](#extraction.transformations.filter.id) | configuration for either File, Values or Regex | Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. | | `is-array` | boolean | Match on whether a node is an array. If this is set to `true` or `false`, the filter will only match variables. | | [`namespace`](#extraction.transformations.filter.namespace) | configuration for either File, Values or Regex | Regex or list of values matching the full namespace of the node ID. | | [`type-definition`](#extraction.transformations.filter.type-definition) | configuration for either File, Values or Regex | Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. | | `historizing` | boolean | Whether the node is historizing. If this is set, the filter will only match variables. | | `node-class` | either `Object`, `ObjectType`, `Variable`, `VariableType`, `DataType` or `ReferenceType` | The OPC UA node class, exact match. | | [`parent`](#extraction.transformations.filter) | object | Another filter instance which is applied to the parent node. | ##### `name` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `DisplayName`. Either one of the following options: * [File](#extraction.transformations.filter.name.file) * [Values](#extraction.transformations.filter.name.values) * [Regex](#extraction.transformations.filter.name.regex) ###### `file` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`name`](#extraction.transformations.filter.name) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `description` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node `Description`. Either one of the following options: * [File](#extraction.transformations.filter.description.file) * [Values](#extraction.transformations.filter.description.values) * [Regex](#extraction.transformations.filter.description.regex) ###### `file` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`description`](#extraction.transformations.filter.description) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `id` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching node ID. IDs on the form `i=123` or `s=string` are matched. Either one of the following options: * [File](#extraction.transformations.filter.id.file) * [Values](#extraction.transformations.filter.id.values) * [Regex](#extraction.transformations.filter.id.regex) ###### `file` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`id`](#extraction.transformations.filter.id) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `namespace` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the full namespace of the node ID. Either one of the following options: * [File](#extraction.transformations.filter.namespace.file) * [Values](#extraction.transformations.filter.namespace.values) * [Regex](#extraction.transformations.filter.namespace.regex) ###### `file` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`namespace`](#extraction.transformations.filter.namespace) configuration. Match field exactly from a list of values Each element of this list should be a string. ##### `type-definition` Part of [`filter`](#extraction.transformations.filter) configuration. Regex or list of values matching the node ID of the type definition. On the form `i=123` or `s=string`. Either one of the following options: * [File](#extraction.transformations.filter.type-definition.file) * [Values](#extraction.transformations.filter.type-definition.values) * [Regex](#extraction.transformations.filter.type-definition.regex) ###### `file` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values read from a local file | Parameter | Type | Description | | - | - | - | | `file` | string | Path to a local file containing a new-line separated list of values. | ###### `values` Part of [`type-definition`](#extraction.transformations.filter.type-definition) configuration. Match field exactly from a list of values Each element of this list should be a string. ### `deletes` Part of [`extraction`](#extraction) configuration. Configure soft deletes. When this is enabled, all read nodes are written to a state store after browse, and nodes that are missing on subsequent browses are marked as deleted from CDF, with a configurable marker. A notable exception is relationships in CDF, which has no metadata, so these are hard deleted if [`cognite.delete-relationships`](#cognite) is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set this to `true` to enable deletes. This requires [`state-storage`](#state-storage) to be configured. | | `delete-marker` | string | Name of marker used to indicate that a node is deleted. Added to metadata, or as a column in Raw. Default value is `deleted`. | ### `status-codes` Part of [`extraction`](#extraction) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | | `ingest-status-codes` | boolean | Whether to ingest status codes into the time series API. | ### `rebrowse-triggers` Part of [`extraction`](#extraction) configuration. Configuration for triggering rebrowse based on changes to specific nodes. | Parameter | Type | Description | | - | - | - | | `namespace-publication-date` | boolean | Indicate if the extractor should subscribe to namespacePublicationDate on the Server node. | | [`namespaces`](#extraction.rebrowse-triggers.namespaces) | list | A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. | #### `namespaces` Part of [`rebrowse-triggers`](#extraction.rebrowse-triggers) configuration. A list of namespace URIs used to filter which `namespace publication data` nodes the extractor will subscribe to. Each element of this list should be a string. ## `events` Global parameter. Configuration for extracting OPC UA events and alarams as CDF events or litedb time series | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | **Required.** Set to `true` to enable reading events from the server | | `history` | boolean | Set to `true` to enable reading event history | | `all-events` | boolean | Set to `true` to read base OPC UA events, in addition to custom events. Default value is `True`. | | `read-server` | boolean | Set to `true` to enable checking the server node when looking for event emitters. Default value is `True`. | | [`destination-name-map`](#events.destination-name-map) | object | Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`.

    If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property.

    **Example:**
    `{'MyProperty': 'SubType', 'MyEndTime': 'EndTime'}` | | [`event-ids`](#events.event-ids) | list | Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. | | `exclude-event-filter` | string | Regex filter on event type `DisplayName`, matches will not be extracted. | | [`emitter-ids`](#events.emitter-ids) | list | List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. | | [`historizing-emitter-ids`](#events.historizing-emitter-ids) | list | List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. | | `discover-emitters` | boolean | Automatically treat nodes with suitable `EventNotifier` as event emitters when they are discovered in the node hierarchy. Default value is `True`. | | [`exclude-properties`](#events.exclude-properties) | list | List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. | ###
    `destination-name-map` Part of [`events`](#events) configuration. Map source browse names to other values in the destination. For CDF, internal properties may be overwritten. By default `Message` is mapped to `description`, `SourceNode` is used to set the `assetIds`, and `EventType` is mapped to `type`. These may be replaced by overrides in `destination-name-map`. If `StartTime`, `EndTime`, or `SubType` are specified, either on the event in OPC-UA itself, or translated through this map, they are set directly on the CDF event. `StartTime` and `EndTime` should be either `DateTime`, or a number corresponding to the number of milliseconds since 01/01/1970. If no `StartTime` or `EndTime` are specified, both are set to the `Time` property. **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} MyProperty: SubType MyEndTime: EndTime ``` | Parameter | Type | Description | | - | - | - | | Any string | either `StartTime`, `EndTime`, `SubType` or `Type` | Target field to map to. | ### `event-ids` Part of [`events`](#events) configuration. Allow-list of event type IDs to map. If this is included, only the events with type equal to one of these node IDs will be included. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `emitter-ids` Part of [`events`](#events) configuration. List of event emitters to extract from. The default behavior is to extract events from nodes based on the `EventNotifier` attribute. This option ignores `EventNotifier` and extracts events from the given list of nodes. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `historizing-emitter-ids` Part of [`events`](#events) configuration. List of node IDs that must be a subset of the `emitter-ids` property. These emitters will have their event history read. The server must support historical events. The `events.history` property must be `true` for this to work. Each element of this list should be a node ID of the data type | Parameter | Type | Description | | - | - | - | | `namespace-uri` | string | Full URI of the node namespace. If left out it is assumed to be the base namespace | | `node-id` | string | Identifier of the node id, on the form i=123, s=string, etc. See the OPC-UA standard | ### `exclude-properties` Part of [`events`](#events) configuration. List of `BrowseName`s of event properties that are to be excluded from automatic mapping to destination metadata. By default, `LocalTime` and `ReceiveTime` are excluded. Be aware that a maximum of 16 metadata entries are allowed in CDF. Each element of this list should be a string. ## `failure-buffer` Global parameter. If the connection to CDF goes down, the OPC UA extractor supports buffering data points and events in a local file or InfluxDB. This is helpful if the connection is unstable, and the server does not provide its own historical data. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable the failure buffer. | | `datapoint-path` | string | Path to a binary file where data points are buffered. Buffering to a file is very fast, and can handle a large amount of data. The file will be created for you if it does not already exist.

    **Example:**
    `buffer.bin` | | `event-path` | string | Path to a binary file for storing events. Buffering to a file is very fast and can handle a large amount of data. The file will be created for you if it does not already exist. Must be different from `datapoint-path`. | | `max-buffer-size` | integer | Set the maximum size in bytes of the buffer file. If the size of the file exceeds this number, no new datapoints or events will be written to their respective buffer files, and any further ephemeral data is lost. Note that if both datapoint and event buffers are enabled, the potential disk usage is twice this number. If set to 0, the buffer file has no upper limit, and will expand until the system is out of storage space, at which point the extractor will probably crash. It is highly recommended to set this if you expect the failure buffer to be used. | | `influx-state-store` | boolean | Set to `true` to enable storing the state of the InfluxDB buffer. This makes the InfluxDB buffer persistent even if the OPC UA extractor stops before it is emptied. Requires [`state-storage`](#state-storage) to be configured. | | `influx` | boolean | Set to `true` to enable buffering data in InfluxDB. This requires a configured influx pusher. See [`influx`](#influx). This serves as an alternative to a local file, but should only be used if pushing to InfluxDB is already an requirement. | | `datapoints-batch` | integer | Maximum number of datapoints to read from the buffer and write to CDF in a single batch. Default value is `1000000`. | | `events-batch` | integer | Maximum number of events to read from the buffer and write to CDF in a single batch. Default value is `10000`. | ##
    `history` Global parameter. Configuration for reading historical datapoints and events from the server | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Set to `true` to enable reading history from the server. If this is `false` no history will be read, overriding configuration set elsewhere. | | `data` | boolean | Set to `false` to disable reading history for data points. Default value is `True`. | | `backfill` | boolean | Enable backfill, meaning that data is read backward and forward through history. This makes it so that the extractor will prioritize reading recent values, then keep reading old data in the background while loading live data. This is potentially useful if there is a lot of history. | | `require-historizing` | boolean | Set this to `true` to require the `Historizing` attribute to be `true` on OPC UA variables in order to read history from them. | | `restart-period` | string | Time in seconds between restarts of history. Setting this too low may impact performance. Leave at 0 to disable periodic restarts. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). This option also allows using cron expressions. | | `data-chunk` | integer | Maximum number of datapoints per variable for each `HistoryRead` service call Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of points to return. Default value is `1000`. | | `data-nodes-chunk` | integer | Maximum number of variables to query in each `HistoryRead` service call. If [`granularity`](#history.granularity) is set, this is applied *after* calculating chunks based on history granularity. Default value is `100`. | | `event-chunk` | integer | Maximum number of events per node for each `HistoryRead` service call for events. Generally, this is limited by the server, so it can safely be set to 0, which means the server decides the number of events to return. Default value is `1000`. | | `event-nodes-chunk` | integer | Maximum number of nodes to query in each `HistoryRead` service call for events. Default value is `100`. | | `granularity` | string | Granularity in seconds for chunking history read operations. Variables with latest timestamp within the same granularity chunk will have their history read together. Reading more variables per operation is more efficient, but if the granularity is set too high, then the same history may be read multiple times. A good choice for this value is `60 * average_update_frequency`.

    Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `600s`. | | `start-time` | string | Earliest timestamp to read history from in milliseconds since January 1, 1970. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Example:**
    `3d-ago` | | `end-time` | string | The latest timestamp history will be read from. In milliseconds since 01/01/1970. Alternatively use syntax `N[timeunit](-ago)` where `timeunit` is one of `w`, `d`, `h`, `m`, `s`, or `ms`. `-ago` indicates that this is set in the past, if left out it will be that duration in the future. | | `max-read-length` | string | Maximum length in time of each history read. If this is greater than zero, history will be read in chunks until the end. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0`. | | `ignore-continuation-points` | boolean | Set to `true` to read history without using `ContinuationPoints`, instead using the `Time` field on events, and `SourceTimestamp` on datapoints to incrementally increase the start time of the request until no data is returned. This is a workaround for server issues, do *not* use this unless you have concrete issues with continuation points. | | [`throttling`](#history.throttling) | object | Configuration for throttling history | | `log-bad-values` | boolean | Log the number of bad datapoints per `HistoryRead` call at Debug level, and each individual bad datapoint at Verbose level. Default value is `True`. | | `error-threshold` | integer | Threshold in percent for a history run to be considered failed. Example: 10.0 -> History read operation would be considered failed if more than 10% of nodes fail to read at some point. Retries still apply, this only applies to nodes that fail even after retries.

    This is safe in terms data loss. A node that has failed during history will not receive state updates from live values, next time history is read, the extractor will continue from where it last successfully read history. Default value is `10`. | ###
    `throttling` Part of [`history`](#history) configuration. Configuration for throttling history | Parameter | Type | Description | | - | - | - | | `max-per-minute` | integer | Maximum number of requests per minute, approximately. | | `max-parallelism` | integer | Maximum number of parallel requests. | | `max-node-parallelism` | integer | Maximum number of concurrent nodes accross all parallel requests. | ## `state-storage` Global parameter. A local LiteDB database or a database in CDF RAW that store various persistent information between extractor runs. This is used to replace reading first/last data points from CDF, and also allows storing first/last times for events. Enabling this is highly recommended, and will be required in a future version of the extractor. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Interval between each write to the buffer file, in seconds. 0 or less disables the state store. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `0s`. | | `variable-store` | string | Name of raw table or litedb collection to store information about extracted OPC UA variables. Default value is `variable_states`. | | `event-store` | string | Name of raw table or litedb collection to store information about extracted events. Default value is `event_states`. | | `influx-variable-store` | string | Name of raw table or litedb collection to store information about variable ranges in the InfluxDB failure buffer. Default value is `influx_variable_states`. | | `influx-event-store` | string | Name of raw table or litedb collection to store information about events in the InfluxDB failure buffer. Default value is `influx_event_states`. | ## `subscriptions` Global parameter. A few options for subscriptions to events and data points. Subscriptions in OPC UA consist of Subscription objects on the server, which contain a list of MonitoredItems. By default, the extractor produces a maximum of four subscriptions: * DataChangeListener - handles data point subscriptions. * EventListener - handles event subscriptions. * AuditListener - which handles audit events. * NodeMetrics - which handles subscriptions for use as metrics. Each of these can contain a number of MonitoredItems. | Parameter | Type | Description | | - | - | - | | [`data-change-filter`](#subscriptions.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | | `data-points` | boolean | Enable subscriptions on datapoints. Default value is `True`. | | `events` | boolean | Enable subscriptions on events. Requires [`events.enabled`](#events) to be set to `true`. Default value is `True`. | | `ignore-access-level` | boolean | Ignore the access level parameter for history and datapoints. This means using the `Historizing` parameter for history, and subscribing to all timeseries. | | `log-bad-values` | boolean | Log bad subscription datapoints. Default value is `True`. | | `keep-alive-count` | integer | The number of publish requests without a response before the server should send a keep alive message. Default value is `10`. | | `lifetime-count` | integer | The number of publish requests without a response before the server should close the subscription. Must be at least 3 \* keep-alive-count. Default value is `1000`. | | `recreate-stopped-subscriptions` | boolean | Recreate subscriptions that have stopped publishing. Default value is `True`. | | `recreate-subscription-grace-period` | string | Grace period for recreating stopped subscriptions. If this is negative, default to 8 \* publishing-interval. Format is as given in [Timestamps and intervals](#timestamps-and-intervals). Default value is `-1`. | | `max-datapoint-notifications-per-publish` | integer | Maximum number of datapoint notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | `max-event-notifications-per-publish` | integer | Maximum number of event notifications per publish. Some servers do not properly support this. The default is 0, which means that the server will decide. | | [`alternative-configs`](#subscriptions.alternative-configs) | list | List of alternative subscription configurations. The first entry with a matching filter will be used for each node. | ###
    `data-change-filter` Part of [`subscriptions`](#subscriptions) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ### `alternative-configs` Part of [`subscriptions`](#subscriptions) configuration. List of alternative subscription configurations. The first entry with a matching filter will be used for each node. Each element of this list should be an alternative subscription configuration | Parameter | Type | Description | | - | - | - | | [`filter`](#subscriptions.alternative-configs.filter) | object | Filter on node, if this matches or is null, the config will be applied. | | [`data-change-filter`](#subscriptions.alternative-configs.data-change-filter) | object | Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | | `sampling-interval` | integer | Requested sample interval per variable on the server. This is how often the extractor requests the server sample changes to values. The server has no obligation to use this value, or to use any form of sampling at all, but according to the standard this should limit the *lowest* allowed rate of updates. 0 tells the server to sample as fast as possible. Default value is `100`. | | `queue-length` | integer | Requested length of queue for each variable on the server. This is how many data points the server will buffer. It should in general be set to at least 2 \* `publishing-interval` / `sampling-interval`

    This setting generally sets the maximum rate of points from the server (in milliseconds). On many servers, sampling is an internal operation, but on some, this may access external systems. Setting this very low can increase the load on the server significantly. Default value is `100`. | ####
    `filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Filter on node, if this matches or is null, the config will be applied. | Parameter | Type | Description | | - | - | - | | `id` | string | | | `data-type` | string | Regex match on node data type, if it is a variable | | `is-event-state` | either boolean or null | Match on whether this subscription is for data points or events | #### `data-change-filter` Part of [`alternative-configs`](#subscriptions.alternative-configs) configuration. Modify the DataChangeFilter used for datapoint subscriptions. See the OPC UA reference part 4 7.17.2 for details. These are just passed to the server, they have no further effect on extractor behavior. Filters are applied to all nodes, but deadband should only affect some, according to the standard. | Parameter | Type | Description | | - | - | - | | `trigger` | either `Status`, `StatusValue` or `StatusValueTimestamp` | What changes to a variable trigger an update. One of `Status`, `StatusValue` or `StatusValueTimestamp`. Default value is `StatusValue`. | | `deadband-type` | either `None`, `Absolute` or `Percent` | Enable deadband for numeric nodes. One of `None`, `Absolute` or `Percent`. Default value is `None`. | | `deadband-value` | integer | Deadband value, effect depends on deadband type. | ## `pub-sub` Global parameter. Configure the extractor to read from MQTT using OPC-UA pubsub. This requires the server pubsub configuration to be exposed through the Server object. You should consider setting subscriptions: data-points: false to avoid duplicate datapoints if this is enabled. | Parameter | Type | Description | | - | - | - | | `enabled` | boolean | Enable PubSub | | `prefer-uadp` | boolean | Prefer using the UADP binary format. If false JSON is preferred. Default value is `True`. | | `file-name` | string | Save or read configuration from a file. If the file does not exist, it will be created from server configuration. If this is pre-created manually, the server does not need to expose pubsub configuration | ## `high-availability` Global parameter. Configuration to allow you to run multiple redundant extractors. Each extractor needs a unique index. | Parameter | Type | Description | | - | - | - | | `index` | integer | **Required.** Unique index of this extractor. Each redundant extractor must have a unique index | | [`raw`](#high-availability.raw) | object | Configuration to use Raw as backend for high availability | | [`redis`](#high-availability.redis) | object | Configuration to use a Redis store as backend for high availability | ### `raw` Part of [`high-availability`](#high-availability) configuration. Configuration to use Raw as backend for high availability | Parameter | Type | Description | | - | - | - | | `database-name` | string | **Required.** Raw database to store high availability states in | | `table-name` | string | **Required.** Raw table to store high availability states in | ### `redis` Part of [`high-availability`](#high-availability) configuration. Configuration to use a Redis store as backend for high availability | Parameter | Type | Description | | - | - | - | | `connection-string` | string | **Required.** Connection string to connect to redis instance | | `table-name` | string | **Required.** Redis table name to store high availability states in | # About the Cognite OSDU extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/osdu Connect to the OSDU Data Platform and ingest data into Cognite Data Fusion (CDF). The features described in this section are in [public preview](/cdf/product_feature_status#public-preview) and may change. The **Cognite OSDU extractor** connects to the Open Group [OSDU™ Data Platform](https://osduforum.org) and ingests data into CDF. Configurable OSDU records, such as wells, wellbores, and well logs, are ingested into the [CDF staging area](/cdf/integration/guides/extraction/raw_explorer) (RAW). Linked data files are streamed to the [CDF files service](/api-reference/concepts/20230101/files). The extractor performs a full data transfer in the first run and keeps polling for changes in OSDU to propagate the data to CDF. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/osdu/osdu_configuration Configuration settings and parameters for the OSDU extractor YAML file. To configure the OSDU extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org) format. The configuration file is split into sections, each represented by a top-level entry in the YAML format. You must name the configuration file *config.yml*. You can use the **sample configuration file** included with the extractor as a starting point for your configuration settings. You must as a minimum adjust these settings before you run the extractor: ```yaml config.yml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} connector: extract: raw-database: Enter the name of the target CDF RAW database. dataset-id: Enter the ID of the target CDF data set (16-digit integer). kinds: Check that this matches the list of OSDU schema IDs to extract. ``` You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ## cognite Include the `cognite` section to configure which CDF project the extractor will load data into and how to connect to the project. This section is mandatory and should always contain the project and authentication configuration. | Parameter | Description | | - | - | | `host` | Insert the base URL of the CDF project. The default value is [https://api.cognitedata.com](https://api.cognitedata.com). | | `project` | Insert the CDF project name you want to ingest data into. | | `timeout` | Specify the number of seconds to wait for a response to a request made to CDF. The default value is 30 seconds. | | `idp-authentication` | Insert the credentials for authenticating to CDF using an external identity provider. | | `client-id` | Enter the client ID from the IdP. | | `scopes` | List the scopes. This is usually \[\{host}/.default]. | | `secret` | Enter the client secret from the IdP. | | `token-url` | Insert the URL to fetch authentication tokens from. | | `extraction-pipeline` | Insert the external ID of an extraction pipeline in CDF. You should create the extraction pipeline before you configure this section. This parameter is optional. | | `external-id` | Enter the external ID of the extraction pipeline in CDF. This parameter is optional. | | `id` | Enter the ID of the extraction pipeline in CDF. This parameter is optional. | ## osdu-client Include the `osdu-client` to configure the connection to the OSDU platform. ### authentication | Parameter | Description | | - | - | | `api-url` | Insert the base URL of the OSDU API. | | `client-id` | Enter the OSDU client ID. | | `client-secret` | Enter the OSDU client secret. | | `data-partition` | Enter the name of the OSDU data partition in OSDU. | | `scope` | Specify the OSDU scope. | | `tenant-id` | Enter the Azure tenant ID. This parameter is optional. | | `timeout` | Enter the maximum time in seconds to wait for a response to a request made to OSDU. The default value is 30 seconds. | | `token-url` | Insert the URL to fetch authentication tokens from. | ### services | Parameter | Description | | - | - | | `cursor-fetch-size` | Specify the number of search results to fetch in each request. | | `dms-parallelism` | Insert the number of parallel threads hitting the DMS API. The default value is 6. | | `generic-parallelism` | Insert the number of parallel threads hitting the generic API. The default value is 32. | ## connector Include the `connector` section to configure the general settings for the extractor. | Parameter | Description | | - | - | | `sleep-time` | Enter the number of seconds to pause between polls for changes. | | `total-parallelism` | Enter the maximum number of threads employed. The default value is 64. | | `upload-queue-size` | Enter the number of items to accumulate in a queue between extraction and writing to CDF RAW and CDF Files. | | `upload-queue-interval` | Enter the number of seconds between uploads of a queue when the size is not reached. | | `extract` | Settings specific to the extraction direction. | | `raw-database` | Enter the database name in CDF RAW that stores the extracted OSDU records. | | `statestore-table` | Enter the name of the auxiliary CDF RAW table that stores the state of the extractor. | | `dataset-id` | Insert the data set ID for the extracted data files. | | `kinds` | List of the OSDU [data types](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/README.md) to extract. Each kind has the following settings: | | `name-pattern` | Enter the name / schema ID of the OSDU kind. You can include multiple kinds in a single entry by entering a pattern with Unix shell-style wildcards (`*` and `?`). For example, `osdu:wks:master-data--Well*:*` would match any version of `osdu:wks:master-data--Well` as well as `osdu:wks:master-data--Wellbore`. | | `filter` | Enter a query following the [Lucene syntax](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html) to filter which records to extract from OSDU, for example `"createTime:[2022-04-19T16 TO *]"` or `"data.Source:\"BLENDED\""`. This parameter is optional. | | `dms-kind` | Use this parameter when the data for the kind is stored in a DDMS instead of generic OSDU files. Supported values are `wellbore_dms_well_log` and `wellbore_dms_trajectory`. This parameter is optional. | ## logger Include the `logger` section to set up logging to a console and files. | Parameter | Description | | - | - | | `console` | Enable logging to a standard output, such as a terminal window. This parameter is optional. | | `level` | Select the verbosity level for console logging. Valid options are `debug`, `info`, `warning`, and `error`. The default value is `info`. | | `file` | Enable logging to a file. This parameter is optional. | | `level` | Select the verbosity level for file logging. Valid options are `debug`, `info`, `warning`, and `error`. The default value is `info`. | | `log_json` | Set to `true` to enable logging in JSON format. The default value is `false`. | | `path` | Insert the file system path to the log file. | | `retention` | Specify the maximum number of days to retain logs. The default value is 7. | ## metrics Include the `metrics` section to send metrics about the extractor performance for remote monitoring of the extractor. This section is optional. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | Parameter | Description | | - | - | | `cognite` | Cognite metrics configurations. This parameter is optional. | | `external_id_prefix` | Enter an external ID prefix to identify the CDF time series created for each metric. | | `push-interval` | Enter the interval in seconds between each push. The default value is 30. | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/osdu/osdu_requirements Hardware and software requirements for running the OSDU extractor. You can install the extractor on any physical server or virtual machine running Windows or Linux. ## Memory * Minimum: 2GB RAM / 20 GB storage. * Recommended: 8GB RAM / 80 GB storage. ## CPU * Minimum: 2-core CPU. * Recommended: 8-core CPU. ## Software * OSDU Data Platform (as a data source). * Optionally, OSDU Wellbore Domain Data Management Services (Wellbore-DDMS). ## Network access * Valid credentials for connecting to the pertinent OSDU instance's API. # Set up the OSDU extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/osdu/osdu_setup Install and configure the OSDU extractor as a Windows executable, Windows service, or Docker container. The Cognite OSDU extractor is available as a Windows executable file (.exe file), a Windows installer (.msi file), and as a Docker container. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/osdu/osdu_requirements). * Assign [access capabilities](/cdf/access/guides/capabilities#osdu-extractor) for the extractor to write data to the respective CDF destination resources. * Set up a [CDF RAW database](/cdf/integration/guides/extraction/raw_explorer#set-up-databases-and-tables-in-cdf-raw) to serve as a target for the records ingested from OSDU. Make a note of the name. * Set up a [data set](/cdf/data_governance/guides/datasets/create_data_sets) to serve as a target for the data files transferred from OSDU. Make a note of the data set ID. * Create an environment file (`.env`) in the folder where you'll run the extractor. This file holds the credentials for connecting to CDF and OSDU. You can use the *.env\_example* file or this template as a starting point: ```bash .env theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} COGNITE_BASE_URL=https://api.cognitedata.com COGNITE_TENANT_ID=1d0fffff-1fff-4fff-bfff-5fffffffffff COGNITE_CLIENT_ID=5a4fffff-3fff-5fff-afff-7fffffffffff COGNITE_CLIENT_SECRET=JZZ1Z~YwZzZzZ~NZz9zzZZ9zZzzz9zzZ_9zZZ COGNITE_TOKEN_URL=https://login.microsoftonline.com/1d0fffff-1fff-4fff-bfff-5fffffffffff/oauth2/v2.0/token COGNITE_CDF_CLUSTER_ID=api COGNITE_PROJECT=name_of_cdf_project OSDU_API_URL=https://api.osduv012.cognite.ai OSDU_TENANT_ID=1d2fffff-8fff-4fff-bfff-6fffffffffff OSDU_CLIENT_ID=f03fffff-9fff-4fff-9fff-2fffffffffff OSDU_CLIENT_SECRET=tPZ9Z~z~9zZZzzZZ9zZzzZ.Z9z_ZzzzZ9Zz9zzZZ OSDU_TOKEN_URL=https://login.microsoftonline.com/1d2fffff-8fff-4fff-bfff-6fffffffffff/oauth2/v2.0/token OSDU_SCOPE=f03fffff-9fff-4fff-9fff-2fffffffffff/.default openid profile offline_access OSDU_PARTITION=example-opendes ``` ## Install the extractor Navigate to **Data fusion** > **Integrate** > **Extractors** and find the OSDU extractor's package for your operating system. Download the zip file and decompress it to the same folder where you created the `.env` file. Create a [configuration file](/cdf/integration/guides/extraction/osdu/osdu_configuration). Open a command line window and run the executable file. You stop the extractor by pressing **Ctrl+C** on your keyboard. The log file is stored in the [configured path](/cdf/integration/guides/extraction/osdu/osdu_configuration#logger). Navigate to **Integrate** > **Extract data** in CDF and find the OSDU extractor installation package for your operating system. Download and install the .msi file. We recommend that you don't change the default installation path. Open the installation folder and edit the configuration file according to the [configuration settings](/cdf/integration/guides/extraction/osdu/osdu_configuration). Set Modify permission under **Properties** in the installation folder to avoid permission issues. Add the environment file (`.env`) you created above to the installation folder. Run *installer.exe* to install the service as an automatic startup service. * To manage the service, run *starter.exe*, *stopper.exe*, and *log\_presenter.exe*. * To remove the extractor, run *remover.exe*. ## Explore the extracted data You can explore the extracted data in [CDF RAW](/cdf/integration/guides/extraction/raw_explorer) and browse the files in the configured CDF data set. ### Data stored in CDF RAW Records ingested from OSDU are stored in the configured CDF RAW database. Adjust and clean up the data before transforming it into the [CDF resource types](/dev/concepts/resource_types). A RAW table is automatically created for each configured OSDU kind, and a row is added for each extracted record. The record ID is used as the row key. Each root key in the record becomes a column. OSDU records displayed in CDF RAW database table ### Data stored in CDF Files The OSDU files linked to the extracted records are saved in [CDF Files](/api-reference/concepts/20230101/files). For instance, this can be CSV, LAS, or LIS files with the row data for a well-log record. You can [transform the data](/cdf/integration/guides/transformation/transformations) to insert the rows into the corresponding [CDF Sequence](/api-reference/concepts/20230101/sequences). The extractor saves the files in CDF as is from the OSDU generic file service API or the DDMS API. The files are not further processed. | Files extracted from OSDU in the [Data explorer](/cdf/integration/guides/exploration/data_explorer) | Files extracted from OSDU in the configured [data set](/cdf/data_governance/guides/datasets/edit_explore_data_sets#explore-data-sets) | | - | - | | OSDU files in CDF Data Explorer interface | OSDU files in CDF Data Catalog interface | ## Monitor extractions Set up [extraction pipelines](/cdf/integration/guides/interfaces/about_integrations) and set the [external ID](/cdf/integration/guides/extraction/osdu/osdu_configuration#cognite) in the configuration file. The first time you run the extractor, it creates the extraction pipeline entries and logs succeeding runs with a success or failed status. You'll see the extraction runs as two sections on the **Run history** tab on the **Extraction pipeline** page. # About Cognite Studio for Petrel extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/petrel_studio Learn about the Cognite Studio for Petrel extractor that streams Petrel object data to Cognite Data Fusion (CDF) as protobuf objects. The Cognite Studio for Petrel extractor connects to [SLB Studio for Petrel](https://www.software.slb.com/products/studio/studio-environment-for-petrel-users) through the [Ocean SDK](https://docs.ocean.dwavesys.com/en/stable/getting_started.html). It streams Petrel object data to the CDF [files service](/api-reference/concepts/20230101/files) as [protobuf](https://developers.google.com/protocol-buffers) objects. The extractor loads data incrementally and only updates and inserts the modified objects as new data revisions to CDF. The extractor is available as a [Windows installer](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_setup). # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration Configuration file settings for the Studio for Petrel extractor including authentication, caching, logging, and extraction parameters. To configure the Studio for Petrel extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org) format. The configuration file is split into sections, each represented by a top-level entry in the YAML format. You can use the [sample minimal configuration file](https://github.com/cognitedata/subsurface-extractor-config-samples/blob/main/petrel-studio-extractor/petrel-studio-extractor-config-sample.yaml) as a starting point for your configuration settings. You can also use substitutions with environment variables in the configuration files. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} `idp-authentication`: project: ${COGNITE_PROJECT} idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} ``` ## Cognite Include the `cognite` section to configure which CDF project the extractor will load data into and how to connect to the project. This section is mandatory and should always contain the project and authentication configuration. | Parameter | Description | | - | - | | `cdf-file-storage` | Set the cloud blob storage engine that will store Petrel data objects. Valid options are `azure` and `google`. | | `cdf-base-url` | Insert the base URL of the CDF project. The default value is \<[https://api.cognitedata.com](https://api.cognitedata.com) >. | | `cdf-project` | Insert the CDF project name you want to ingest data into. | | `cdf-oidc-authority` | Insert the credentials for authenticating to CDF using an external identity provider. You must enter either an API key or use IdP authentication. | | `cdf-oidc-client-id` | Enter the client ID from the IdP. | | `cdf-oidc-secret` | Enter the client secret from the IdP. | | `cdf-oidc-tenant` | Enter the Azure tenant. | | `cdf-oidc-scopes` | List the scopes. | | `cdf-raw-database` | Enter the CDF RAW database name that will store Studio for Petrel objects state store. If the database doesn't already exist, the extractor creates a database. | | `data-set-id` | Enter the data set ID for the CDF Files. Set to 0 or no value if you don't want to add a data set ID. | | `state-db` | Select which database to store the extractor state. If you don't use this parameter, the extractor uses an in-memory state store. | | `api-version-cache` | Before the extractor serializes a record from Studio for Petrel, a call goes to the API to verify if the object has been loaded. This cache holds the last state for quicker restarts. The default value is state\api-version-cache.dat" | ## Cache Include the `cache` section to enable or disable caching of data to disk. Use these parameters to avoid intermittent failures. | Parameter | Description | | - | - | | `enable-cache` | Set to `true` to enable caching of data to disk and avoid intermittent failures. | | `cache-location` | Set the path to store cached objects. You can specify this as

  • C:\cache - fully qualified
  • cache - relative to the installation folder.

  • If the remaining disk space is less than 10 GB, caching is automatically turned off. You need *Modify* permissions on the folder. | ## Logging Include the `logging` section to set up logging to a standard output, such as a terminal window. | Parameter | Description | | - | - | | `log-to-console` | Set to `true` to log to a console. This parameter is used if you're running the extractor from a terminal, such as Powershell. | | `log-to-file` | Set to `true` to log to a file. | | `log-filename` | Add a prefix string in the name for the generated log files. This is an optional parameter. | | `retention` | Specify the maximum number of days the log files are retained in the log folder. | | `level` | Select the verbosity level for logging. Valid options are `verbose`, `debug`, `information`, `warning`, `error`, `fatal`. The default value is `information`. | ## Petrel-studio Include the `petrel-studio` section to configure the connection to Studio for Petrel. | Parameter | Description | | - | - | | `server` | Insert the hostname or IP address of the Studio for Petrel server. | | `license-server` | If you're running the extractor as a Windows service, uncomment and specify the license server if the environment variable `SLBSLS_LICENSE_FILE` isn't set or unavailable. | | `Provider` | Enter the name of the database type, such as *Oracle* or *SQL Server*. The default value is *SQL Server*. | | `Port` | Insert the SQL server port. The default value is 1433. | | `use-integrated-security` | Set to `true` to use Windows integrated security to authenticate to Studio for Petrel server. The default value is `false`.
    On SQL Server based repositories, accessing Petrel Studio data types that use filestreams (i.e. surface, horizon, pointset) requires this option to be enabled and the user and machine must be joined to the same Active Directory domain as the Petrel Studio instance. | | `username` | Enter the username to Studio for Petrel if you're not using integrated security. | | `password` | Enter the password to Studio for Petrel if you're not using integrated security. | | `data-source-name` | Insert the Studio for Petrel data source name. | | `service-name` | Enter the service name if this is specified in the data source. | | `node-id` | Node ID - string. The default value is 1. | | `throttle-time` | Insert the time in milliseconds to wait between pulling data. The default value is 1000. | | `operation-mode` | Select an operation mode:
    1. Extract all objects, but limited by `primary-batch-size`. This is the default value.
    2. Extract only objects with business tags or a valid data status but limited by `primary-batch-size`.
    3. Extract first objects with business tags or a valid data status, limited by `primary-batch-size`, then extract any objects limited by `secondary-batch-size`. | | `primary-batch-size` | Limit the number of objects to extract from an object type before moving to the next type. The valid range is from 0 to 10000. The default value is 0, indicating no limit. | | `secondary-batch-size` | Limit the number of objects to extract from an object type before moving to the next type. The valid range is from 0 to 10000. The default value is 0, indicating no limit. | | `connector-instance` | Enter the connector instance name or ID to configure studio repositories to extract data from on the API side. If you set repositories in configuration here, it overrides the configured repositories in the API. | | `repositories` | Enter the Studio for Petrel repositories to extract data from. | | `data-types` | List the Petrel data types to be sent to CDF. An example list is available [here](https://github.com/cognitedata/subsurface-extractor-config-samples/blob/805f31fa526df50d56e74897d2b11b01636f1c7f/petrel-studio-extractor/petrel-studio-extractor-config-sample.yaml#L142). | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/petrel_studio/petrel_studio_requirements Hardware, software, and network requirements for installing the Studio for Petrel extractor. You can install the extractor on a Windows physical server or a virtual machine on-premises alongside the Petrel environment. ## Memory * Minimum: 8GB RAM / 100 GB storage. * Recommended: 16GB RAM / 100 GB storage (intensive extractions / considerable amount of historical records). ## CPU * Minimum: 4-core CPU. * Recommended: 6-core CPU (intensive extractions / considerable amount of historical records). ## Software * Supported version: Studio for Petrel 2020. * Windows Server 2016 or later. * Studio Runtime 2020. * .NET Framework 4.8.0 or later. ## Network access * User or service account with access to Studio for Petrel data repositories. * A valid STUDIO\_RUNTIME client license, included in Studio for Petrel environment. # Set up the Studio for Petrel extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/petrel_studio/petrel_studio_setup Install and configure the Cognite Studio for Petrel extractor as a Windows service. The Cognite Studio for Petrel extractor is distributed as a Windows installer (.msi file). When you run the .msi file, a setup wizard guides you through the installation process. The installer creates a Windows service and installs the binary files needed to run the extractor. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_requirements) for the extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#studio-for-petrel-extractor) in a Cognite Data Fusion (CDF) project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). * Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime. ## Install and configure the extractor Navigate to Data fusion > Integrate > Extractors and download and run the latest Studio for Petrel extractor .msi file. In the installation folder, you'll also find log files, caches, the configuration file, and local state databases to keep track of which data that's transferred. All relative paths you see in these articles are relative to this folder. **Set modify permissions**: Set Modify permission under Properties in the installation folder to avoid permission issues. Create a `config` folder within the installation folder. Create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration). The file must be in YAML format. ## Run as a Windows service Start the Studio for Petrel extractor using the Windows service or with a command line: ```yml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} net start ConnectorService ``` The Windows service is named **Connector**. Check that the service is running properly. You should see the configuration directory structure: Configuration directory The logs from the extractor startup depend on the configuration but are similar to: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [02:43:06.667 INF] Setup to read/persist list of last (object, version) checked against the API from/to 'state\api-version-cache.dat' [02:43:09.782 INF] Preheating API version cache from state\api-version-cache.dat [02:43:09.797 INF] Preheating cache with 14734 entries [02:43:10.332 INF] Compatible Studio server installation found. [02:43:20.716 INF] Repositories available on server petrel-studio-001.domain: COG_DEV, COG_INTEGRATION_TEST, TEST_CDF_DATA_XFER [02:43:25.933 INF] Persisting objects to remote API for COG_DEV. [02:43:25.933 INF] Processing repository 'COG_DEV' ``` The extractor sequentially checks all configured [repositories](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration#petrel-studio). Select the interval between iterations of rechecking repositories with the [`throttle-time`](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration#petrel-studio) configuration setting. When stopping the service, this is logged as: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [02:42:49.057 INF] Windows service signalled stop, stopping extractor. [02:42:49.217 INF] Releasing licence. [02:42:49.257 INF] Licence released. ``` ## Run from a command line You can run the extractor from a command line. Navigate to the installation folder in a command line window. Start the command line window as the user that normally runs the service. This triggers latent permission issues and avoids local permission issues when you want to create files inside the installation folder. Run *Connector.exe*. The extractor reads and configures the extractor according to the configuration settings. If you've configured the `log to file` parameter, the extractor logs to the command line window and the matching log under the `logs` folder. ## Check the extracted data The extractor saves Studio for Petrel records as protobuf objects in CDF files. The records are loaded incrementally since the extractor checks for changes in Studio for Petrel and only sends the modified records to CDF. Extracted Records ### Internal tables The extractor uses two internal tables for execution configuration and monitoring: * `StudioRepos`: Configuration table that stores Studio for Petrel `instances` and the list of repositories to be extracted inside `studioRepositoryName`. The list of repositories is the same as the list configured in the YAML file. * StudioRepositoryName: This table is automatically created using the values set in `studioRepositoryName`. For example, if you have configured `Gullfaks` as the repository to be extracted, a table called `Gullfaks` is created in the RAW database. This table stores the data revisions and timestamp of the last extraction for the related Studio for Petrel objects. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/extraction/petrel_studio/petrel_studio_troubleshooting Resolve errors and unexpected behavior with the Cognite Studio for Petrel extractor. ## View error messages You'll find information related to the Windows service under **Windows Logs** > **Application** in the **Event Viewer**. To log messages from the extractor, configure the parameters in the [Logging](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration#logging) configuration section. The log file is named `log-Year-Month-Day.log`. By default, this log is stored in the logs directory within the installation directory. The log file rotates once a day and is kept for 7 days. You can change this with the [`retention`](/cdf/integration/guides/extraction/petrel_studio/petrel_studio_configuration#logging) configuration parameter. Use [Powershell's Get-Content -Tail](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-content?view=powershell-7.3) to monitor changes in the logs. ## Failed runs If the extractor fails to run, stop the Windows service, and start `Connector.exe` from the command line. Note that the command prompt must run as the same user as the Windows service to get the same Studio for Petrel access. ## Configuration errors Configuration errors throw exceptions when the extractor runs. You'll find the missing or incorrect parameters in the execution logs. When you've fixed the configuration file, restart the extractor service: Stop the extractor service execution by running: ```yml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} net stop ConnectorService ``` Start the extractor service execution by running: ```yml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} net start ConnectorService ``` If a service is constantly restarting due to errors, the Windows server may choose to throttle the service instead of restarting. You may need to start the service manually using the Windows service. ### Windows events log You can use the Windows Event Viewer to troubleshoot errors when the extractor runs. Issues such as incorrect credentials when starting the service won't be logged as files, so you must check the Event Viewer logs. Use the event logs in combination with the log file to debug errors or find incorrect configuration settings. # About the Cognite PI extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi Connect to OSISoft PI Data Archive and stream time series data to Cognite Data Fusion (CDF) in real-time with historical backfill support **Deploy with the Cognite Toolkit:** The [Foundation deployment pack](/cdf/deploy/cdf_toolkit/references/packages/foundation) includes a `cdf_pi_extractor` module with ready-made extraction pipeline configs — use it instead of configuring the extractor pipeline by hand. The **PI points** in the **PI Data Archive** correspond to the time series in CDF. ## How the extractor processes data The Cognite PI extractor extracts the time series data from the PI Data Archive according to your filters, creates corresponding time series in CDF, and then feeds all PI points to the time series in CDF. The extractor supports CDF [data sets](/cdf/data_governance/concepts/datasets), and each extractor can push data into one data set. In CDF, the time series get an external ID composed of a configurable prefix and the PI point *name*. You can configure the extractor to use the PI point *ID* if the PI points don't have unique names. ### Extraction modes The Cognite PI extractor runs one or more of these tasks concurrently: * **Backfill** - Fills in historical data backward from the **first** data point in CDF. * **Frontfill** - Fills in historical data forward from the **last** data point in CDF. * **Streaming** - Adds data points to CDF in **real-time**. The **first time** you run the extractor, it creates all the necessary time series in CDF, streams data points from the current time, and backfills the time series with historical data. If the extractor **reruns** after a period of downtime, it resumes the backfill task and starts a frontfill task to fill in the gap between when the extractor stopped and the current time. When the frontfill task has caught up, the extractor returns to streaming live data points. The extractor maintains an extraction state for the time range between the first and last data point inserted into CDF. Only the streaming task can insert data points in CDF within this range. Any changes to historical values already existing in CDF will only be updated in CDF when the extractor is streaming data. The PI extractor can miss data updates on the PI server. For instance, if the extractor restarts while the PI server gets historical updates. Changes in PI within a time range that has already been extracted is picked up by streaming if the PI extractor is running while the change is made. If the PI extractor isn't running while the change is made, the change isn't replicated to CDF. The backfill and frontfill tasks extract **compressed** data points from the PI Data Archive, while the streaming task extracts **all** data points. The compression settings of your PI system may impact the data quality for your data models. CDF receives data points from streaming that may be later removed in PI by compression. For this reason, there may be more data points in CDF than in PI. ### Time series processing The extractor runs an **update** task on a schedule, by default, every 24 hours and when it reconnects to the PI server after downtime. The update task discovers new PI Points and adds the corresponding time series in CDF, including the metadata. If a PI Point has been deleted on the PI server, the extractor doesn't remove the associated time series in CDF. The extractor logs the deletion and keeps track of the number of deleted PI Points. The extractor resets the number when it restarts or reconnects to the PI server. This is the mapping between PI Point attributes (metadata) and fields to time series attributes and metadata in CDF: | Source (PI Point) | Destination (CDF) | | - | - | | `engunits` | `Unit` | | `descriptor` | `Description` | | `PointType` | `IsString`

    The `PIPointType` maps to numeric (`IsString` = `false`) for `Digital`, `Float16/32/64`, `Int16/32` and `Timestamp`.

    All other types (including `String`) maps to `IsString` = `true`. | | `Name` | `Name` | | Attributes (filters below) | `MetaData`, every value converted to string | | `` + `Name` | `ExternalId`, `LegacyName` | | `step` attribute equal to 1 | `IsStep` | Internal bookkeeping attributes in the PI system aren't included in the CDF time series metadata: `creationdate`, `creator`, `changedate`, `changer`, `ptowner`, `ptgroup`, `ptaccess`, `ptsecurity`, `dataowner`, `datagroup`, `dataaccess`, and `datasecurity`. ### Data points processing Each data point in the source has a source timestamp, a generic value, and a `IsGood` flag. The extractor sets the source timestamp in PI as the data point timestamp in CDF. Because the timestamp resolution in PI is higher than in CDF (milliseconds), multiple data points in PI can be mapped to the same timestamp in CDF. In the unlikely event that this should happen, the PI extractor can't guarantee which data point it extracts to CDF. The **`IsGood`** flag in PI indicates that there is an issue reading the value from the sensor or control system. In these cases, the PI value may be a string description even though the PI Point has a numeric type. The extractor ignores data points with `IsGood = false`. For `IsGood = true`, it tries to convert the value to the CDF time series type. If the conversion fails, typically only for numeric time series in CDF, the data point is ignored. The extractor logs the number of ignored data points. The extractor ignores PI's unit of measure (UOM) property per data point. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/4-20 PI extractor configuration reference for version 4.20. This page documents PI extractor **4.20**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/pi/pi). To configure the PI extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org/) format. The configuration file is split into [sections](#configure-the-pi-extractor), each represented by a top-level entry in the YAML format. Subsections are nested under a section in the YAML format. You can use either the sample complete or minimal configuration files included with the [installer](../pi/pi_setup.md) as a starting point for your configuration settings: * config.default.yml - This file contains all configuration options and descriptions. * config.minimal.yml - This file contains a minimum configuration and no descriptions. You must name the configuration file *config.yml*. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Before you start * Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`. * The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway. * Set up an [extraction pipeline](../../interfaces/add_integrations.md) and note the external ID. ## Minimal YAML configuration file The YAML settings below contain valid PI extractor 2.1 configurations. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file has a global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 3 of the configuration schema. ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 3 cognite: project: '${COGNITE_PROJECT}' idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} time-series: external-id-prefix: 'pi:' pi: host: ${PI_HOST} username: ${PI_USER} password: ${PI_PASSWORD} state-store: database: LiteDb location: 'state.db' logger: file: level: 'information' path: 'logs/log.log' ``` where: * `version` is the version of the configuration schema. Use version 3 to be compatible with the Cognite PI extractor 2.1. * `cognite` is the how the extractor reads the authentication details for PI (pi) and CDF (cognite) from environment variables. Since no host is specified in the `cognite` section, the extractor uses the default value, \<[https://api.cognitedata.com](https://api.cognitedata.com) >, and assumes that the PI server uses Windows authentication. * `time-series` configures the extractor to create time series in CDF where the external IDs will be prefixed with `pi:`. You can also use a data set ID configuration to add all time series created by the extractor to a particular data set. * `state-store` configures the extractor to save the extraction state locally using a LiteDB database file named `state.db`. * `logger` configures the extractor to log at information level and outputs log messages to a log file in the logs/log.log directory. By default, new files are created daily and retained for 31 days. The date is appended to the file name. ## Writing to CDF Data Models The extractor can write directly to a [core data model time series](../../../../dm/dm_reference/dm_core_data_model#timeseries), or more correctly, to an extended version of the `CogniteTimeSeries` Data Model type. To save/update instances of `CogniteTimeSeries`, specify the [identifier for the target Data Modeling Space](https://api-docs.cognite.com/20230101/tag/Spaces/operation/ApplySpaces#!path=items/space\&t=request) in the configuration, using the `space-id` parameter of the [time series configuration](#-time-series) section. The behavior in this mode is almost identical to classic time series destination, except for a few key differences: * Metadata is written as a json blob to the extended time series type (`CogniteExtractorTimeSeries`), and saved in the `extractedData` property. * `data-set-id` configuration option is ignored. ## Using values from Azure Key Vault The PI extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} key-vault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ##
    Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For configuring the earliest point to backfill to you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the backfill only recent history. You can also set the backfill target to a specific date on RFC 3339 form, like `2025-07-02T22:23:12Z`, or on a shorter form like `2022-11-20+12:00`. A timezone specifier is necessary, `Z` means UTC. ## Configure the PI extractor Configuration for the PI extractor. Each section configures a different aspect of the extractor. | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | | [`metrics`](#metrics) | object | Configuration for publishing metrics. | | [`cognite`](#cognite) | object | Configure connection to Cognite Data Fusion (CDF) | | [`state-store`](#state-store) | object | Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | | [`pi`](#pi) | object | Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | | [`time-series`](#time-series) | object | Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. | | [`events`](#events) | object | Configuration for writing events on reconnect and data loss incidents | | [`extractor`](#extractor) | object | The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:

    1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
    2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | | [`backfill`](#backfill) | object | Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | | [`frontfill`](#frontfill) | object | Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. | | [`high-availability`](#high-availability) | object | Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | ##
    `logger` Global parameter. Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | ### `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configuration for publishing metrics. | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. | | [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for having the extractor start a Prometheus scrape server on a local port. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.

    **Examples:**
    `localhost`
    `0.0.0.0` | | `port` | integer | **Required.** The port used for a local Prometheus server. | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://my.pushgateway:9091` | | `job` | string | **Required.** Name of the Prometheus pushgateway job. | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ##
    `cognite` Global parameter. Configure connection to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`metadata-targets`](#cognite.metadata-targets) | object | Configuration for targets for time series metadata. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configuration for targets for time series metadata. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Configuration for writing metadata to CDF Raw. | | [`clean`](#cognite.metadata-targets.clean) | object | Configuration for enabling writing metadata to CDF Clean. | #### `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for writing metadata to CDF Raw. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The Raw database to write to. | | `timeseries-table` | string | **Required.** Name of the Raw table to write timeseries metadata to, enables writing metadata to Raw. Metadata in this case includes name, description, and unit. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for enabling writing metadata to CDF Clean. | Parameter | Type | Description | | - | - | - | | `timeseries` | boolean | Set to `false` to disable writing metadata to time series. Default value is `True`. | ## `state-store` Global parameter. Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Enter the time between each write to the state store. 0 or less disables the state store. Format is as given in [Intervals](#intervals). Default value is `10s`. | | `time-series-table-name` | string | Table name in Raw/Redis/LiteDB for storing time series state. Default value is `ranges`. | | `extractor-table-name` | string | Table name in Raw/Redis/LiteDB for storing general extractor state. Default value is `extractor`. | | `redis` | boolean | Use a redis state store. Overrides `database`. | ## `pi` Global parameter. Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Enter the hostname or IP address of the PI server or the PI collective name. If the host is a collective member name, connect to that member. | | `username` | string | Enter the username to use for authentication. Leave this empty if the Windows service account under which the extractor runs is authorized in Active Directory to access the PI host. | | `password` | string | Enter the password for the given username, if any. | | `native-authentication` | boolean | Determines whether the extractor will use native PI authentication or Windows authentication. The default value is `false`, which indicates Windows authentication. | | `parallelism` | integer | Insert the number of parallel requests to the PI server. If `backfill-parallelism` is set, this excludes backfill requests. Default value is `1`. | | `backfill-parallelism` | integer | Insert the number of parallel requests to the PI server for backfills. This allows the separate throttling of backfills. The default value is 0, which means that the value in `parallelism` is used. | | `use-member-priority` | boolean | When connecting to a PI Collective, attempt to connect to the member with highest priority. If set to `false`, attempt to connect to the member with the same name as `host`. | | `max-connection-retries` | integer | The maximum number of times to attempt to connect to the PI server before failing fatally. If this is `0`, retry forever. | ## `time-series` Global parameter. Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. The example would create time series on the following form in CDF: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi:12345", "name": "PI-Point-Name", "isString": false, "isStep": false, "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} external-id-prefix: 'pi:' external-id-source: SourceId data-set-id: 1234567890123456 ``` | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | Enter the external ID prefix to identify the time series in CDF. Leave empty for no prefix. The external ID in CDF will be this prefix followed by either the PI Point name or PI Point ID. | | `external-id-source` | either `Name` or `SourceId` | Enter the source of the external ID. `Name` means that the PI Point name is used, while `SourceID` means that the PI Point ID is used. Default value is `Name`. | | `data-set-id` | integer | Specify the data set to assign to all time series controlled by the extractor, both new and current. If you don't configure this, the extractor will not change the current time series' data set. | | `data-set-external-id` | string | Specify the external ID of the data set to use, see `data-set-id`. Using this requires the dataSets:READ ACL in CDF | | `sanitation` | either `Remove`, `Clean` or `None` | Specify what to do when the time series fields exceed CDF limits. `Remove` will skip any time series that fail sanitation. `Clean` will truncate and remove values to conform to limits. `None` does nothing (requests may fail as a result). External IDs are never truncated, any time series exceeding CDF limits will be skipped to avoid external ID collisions, regardless of this configuration. Default value is `Clean`. | | `update-metadata` | boolean | Enable updating time series if extractor configuration or PI metadata changes. Default value is `True`. | | `space-id` | string | The identifier for the Data modeling space. This extractor configuration parameter overrides `data-set-id` parameter, and changes the extractor destination from the Asset-Centric TimeSeries to a Core Data Model based TimeSeries, using the `CogniteExtractorTimeSeries` system model extension. | | `source-id` | string | Data modeling source id, this parameter will override the external id of the source node in data modeling, which defaults to the `host` parameter from PI server configuration. | ## `events` Global parameter. Configuration for writing events on reconnect and data loss incidents The example configuration produces events on the following form ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi-events:PiExtractor-2020-08-04 01:01:21.395(0)", "startTime": 1596502850668, "endTime": 1596502880393, "type": "DataLoss", "subtype": "DataPipeOverflow", "source": "PiExtractor", "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} source: PiExtractor external-id-prefix: 'pi-events:' data-set-id: 1234567890123456 store-extractor-events-interval: 5m ``` | Parameter | Type | Description | | - | - | - | | `source` | string | Events have this value as the `source` in CDF. | | `external-id-prefix` | string | Set an external ID prefix to identify events in CDF. Leave empty for no prefix. | | `data-set-id` | integer | Select a data set to assign to all events created by the extractor. We recommend using the same data set ID used in the [Time series](#time-series) section. | | `data-set-external-id` | string | Data set external ID added to events written to CDF, see `data-set-id`. Using this requires the `dataSets:READ` ACL in CDF | | `store-extractor-events-interval` | string | Store events in CDF with this interval. Format is as given in [Intervals](#intervals). If this is not set, events are not created in CDF.

    **Examples:**
    `10s`
    `5m` | ##
    `extractor` Global parameter. The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created: 1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points. 2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | Parameter | Type | Description | | - | - | - | | `attribute-filter-case-insensitive` | boolean | Enable case-insensitive comparison for attribute keys and values in include-attribute-values and exclude-attribute-values filters. When enabled, attribute value 'ABC' will match 'abc', 'Abc', 'ABC', etc. NOTE: This setting does NOT affect tag name matching (include-tags, exclude-tags), prefix matching (include-prefixes, exclude-prefixes), or pattern matching (include-patterns, exclude-patterns), which remain case-sensitive. Default is false (case-sensitive matching). | | [`include-tags`](#extractor.include-tags) | list | Include tags with name exactly equal to one of these names. | | [`include-prefixes`](#extractor.include-prefixes) | list | Include tags with name starting with one of these prefixes. | | [`include-patterns`](#extractor.include-patterns) | list | Include tags with name containing one of these substrings. | | [`include-attribute-values`](#extractor.include-attribute-values) | list | Include tags with attributes equal to at least one of these key/value pairs. | | [`exclude-tags`](#extractor.exclude-tags) | list | Exclude tags with name exactly equal to one of these names. | | [`exclude-prefixes`](#extractor.exclude-prefixes) | list | Exclude tags with name starting with one of these prefixes. | | [`exclude-patterns`](#extractor.exclude-patterns) | list | Exclude tags with name containing one of these substrings. | | [`exclude-attribute-values`](#extractor.exclude-attribute-values) | list | Exclude tags with attributes equal to at least one of these key/value pairs. | | `time-series-update-interval` | either string or integer | Interval between PI tag scans. Each scan discovers new PI tags and detects tags removed from PI but still present in CDF; removed tags are handled per extractor.deleted-time-series.behavior (applies to both classic and CDM time series, plus RAW metadata when enabled). The accepted formats for this field is either in the form of intervals like `4d`, or in the form of a cron expression like `0 4 * * *`. Default value is `24h`. | | [`deleted-time-series`](#extractor.deleted-time-series) | object | Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). | | `end-of-stream-interval` | string | Interval for fetching the end-of-stream timestamps from PI. Format is as given in [Intervals](#intervals). | | `end-of-stream-chunking` | integer | Maximum number of time series per end-of-stream request. Default value is `10000`. | | `end-of-stream-throttle` | integer | Maximum number of parallel end-of-stream requests. Default value is `10`. | | `time-series-update-throttle` | integer | Maximum number of parallel time series updates. Default value is `10`. | | `dry-run` | boolean | Set this to `true` to run the extractor in dry-run mode, where it does not push any data to CDF. Useful for testing the connection to the PI Server | | `read-extracted-ranges` | boolean | Set this to `false` to disable reading from CDF on extractor startup. If this is set to `false`, it is strongly recommended to have a state-store configured, or the extractor will read *all* history from PI on every run. Default value is `True`. | | [`status-codes`](#extractor.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | ### `include-tags` Part of [`extractor`](#extractor) configuration. Include tags with name exactly equal to one of these names. Each element of this list should be a string. ### `include-prefixes` Part of [`extractor`](#extractor) configuration. Include tags with name starting with one of these prefixes. Each element of this list should be a string. ### `include-patterns` Part of [`extractor`](#extractor) configuration. Include tags with name containing one of these substrings. Each element of this list should be a string. ### `include-attribute-values` Part of [`extractor`](#extractor) configuration. Include tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `exclude-tags` Part of [`extractor`](#extractor) configuration. Exclude tags with name exactly equal to one of these names. Each element of this list should be a string. ### `exclude-prefixes` Part of [`extractor`](#extractor) configuration. Exclude tags with name starting with one of these prefixes. Each element of this list should be a string. ### `exclude-patterns` Part of [`extractor`](#extractor) configuration. Exclude tags with name containing one of these substrings. Each element of this list should be a string. ### `exclude-attribute-values` Part of [`extractor`](#extractor) configuration. Exclude tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `deleted-time-series` Part of [`extractor`](#extractor) configuration. Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). This only affects time series with the same data set ID and external ID prefix as the [time series configured in the extractor](#time-series) To find time series that exist in CDF but not in PI, the extractor: * Lists all time series in CDF that have the configured external ID prefix and data set ID * Filters the time series using the include/exclude rules defined in the [extractor](#extractor) section. * Matches the result against the time series obtained from the PI Server after filtering these using the include/exclude rules. | Parameter | Type | Description | | - | - | - | | `behavior` | either `None`, `Flag` or `Delete` | Select the action taken by the extractor. Setting this to `Flag` will perform soft deletion: Flag the time series as deleted but don't delete them from CDF. Setting it to `Delete` will delete the time series from CDF.

    **If you set this to `Delete`, the time series in CDF that cannot be found in PI will be permanently deleted from CDF.**. Default value is `None`. | | `flag-name` | string | If you've set `behavior` to `Flag`, use this string to mark the time series as deleted. Metadata is added to the time series with this as the key, and the current time as the value. Default value is `deletedByExtractor`. | | `time-series-delete-throttle` | integer | Maximum number of parallel delete operations. Default value is `10`. | ###
    `status-codes` Part of [`extractor`](#extractor) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | ## `backfill` Global parameter. Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set to `true` to disable backfill. | | `step-size-hours` | integer | Step, in whole number of hours. Set to 0 to freely backfill all time series. Each iteration of backfill will backfill all time series to the next step before stepping further backward. This helps even out the progression of long backfill processes. Default value is `168`. | | `to` | string | The target CDF timestamp in miliseconds at which to stop the backfill. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Examples:**
    `3d-ago`
    `2025-07-02T22:23:12Z`
    `2022-11-20+12:00` | | `time-series-delay` | integer | Delay in milliseconds between each time series backfill request within a step. | | `step-delay` | integer | Delay in milliseconds between each step. | | `retry-bucket-size` | integer | Maximum size of the retry bucket. Zero or less for no limit. Default value is `100`. | | `retry-bucket-cleanup-time` | integer | Delay in seconds between each cleanup of the retry bucket. Default value is `3600`. | ##
    `frontfill` Global parameter. Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. When data points are archived in PI, they may be subject to compression, reducing the total amount of data points in a time series. Therefore, the backfill and frontfill tasks will receive data points after compression, while the streaming task will receive data points before compression. Learn more about compression in [this video](https://www.youtube.com/watch?v=6-scv3oQ7Kk). | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set this to `true` to disable frontfill and streaming. | | `streaming-interval` | string | Interval between each call to the PI Data Pipe to fetch new data. Format is as given in [Intervals](#intervals). If set to zero, the extractor will not fetch live data from PI.

    **If you set this parameter to a high value, there is a higher chance of having a client outbound queue overflow in the PI server. Overflow may cause loss of data points in CDF.**

    Recommended range is between 1 and 30 seconds. If you observe data loss events, consider turning this value down. Default value is `1s`. | | `delete-data-points` | boolean | If you set this to `true`, the corresponding data points are deleted in CDF when data point deletion events are received in the PI Data Pipe.

    **Enabling this parameter may increase the streaming latency of the extractor since the extractor verifies the data point deletion before proceeding with the insertions.** | | `use-data-pipe` | boolean | Older PI servers may not support data pipes. If that's the case, set this value to `false` to disable data pipe streaming. The frontfiller task will run constantly and will frequently query the PI Data Archive for new data points. Default value is `True`. | | `time-series-chunk` | integer | The maximum number of time series in each frontfill query to PI. Value of this parameter cannot be larger than the value of `data-points-chunk`. Default value is `1000`. | | `data-points-chunk` | integer | The maximum number of requested data points in each frontfill query to PI. Default value is `10000`. | ##
    `high-availability` Global parameter. Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | Parameter | Type | Description | | - | - | - | | `interval` | string | Interval to update the high availability state in Redis. Format is as given in [Intervals](#intervals). | | `timeout` | integer | Timeout in seconds before taking over as active extractor. Must be set greater than 0 to enable high availability. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/4-21 PI extractor configuration reference for version 4.21. This page documents PI extractor **4.21**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/pi/pi). To configure the PI extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org/) format. The configuration file is split into [sections](#configure-the-pi-extractor), each represented by a top-level entry in the YAML format. Subsections are nested under a section in the YAML format. You can use either the sample complete or minimal configuration files included with the [installer](../pi/pi_setup.md) as a starting point for your configuration settings: * config.default.yml - This file contains all configuration options and descriptions. * config.minimal.yml - This file contains a minimum configuration and no descriptions. You must name the configuration file *config.yml*. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Before you start * Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`. * The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway. * Set up an [extraction pipeline](../../interfaces/add_integrations.md) and note the external ID. ## Minimal YAML configuration file The YAML settings below contain valid PI extractor 2.1 configurations. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file has a global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 3 of the configuration schema. ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 3 cognite: project: '${COGNITE_PROJECT}' idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} time-series: external-id-prefix: 'pi:' pi: host: ${PI_HOST} username: ${PI_USER} password: ${PI_PASSWORD} state-store: database: LiteDb location: 'state.db' logger: file: level: 'information' path: 'logs/log.log' ``` where: * `version` is the version of the configuration schema. Use version 3 to be compatible with the Cognite PI extractor 2.1. * `cognite` is the how the extractor reads the authentication details for PI (pi) and CDF (cognite) from environment variables. Since no host is specified in the `cognite` section, the extractor uses the default value, \<[https://api.cognitedata.com](https://api.cognitedata.com) >, and assumes that the PI server uses Windows authentication. * `time-series` configures the extractor to create time series in CDF where the external IDs will be prefixed with `pi:`. You can also use a data set ID configuration to add all time series created by the extractor to a particular data set. * `state-store` configures the extractor to save the extraction state locally using a LiteDB database file named `state.db`. * `logger` configures the extractor to log at information level and outputs log messages to a log file in the logs/log.log directory. By default, new files are created daily and retained for 31 days. The date is appended to the file name. ## Writing to CDF Data Models The extractor can write directly to a [core data model time series](../../../../dm/dm_reference/dm_core_data_model#timeseries), or more correctly, to an extended version of the `CogniteTimeSeries` Data Model type. To save/update instances of `CogniteTimeSeries`, specify the [identifier for the target Data Modeling Space](https://api-docs.cognite.com/20230101/tag/Spaces/operation/ApplySpaces#!path=items/space\&t=request) in the configuration, using the `space-id` parameter of the [time series configuration](#-time-series) section. The behavior in this mode is almost identical to classic time series destination, except for a few key differences: * Metadata is written as a json blob to the extended time series type (`CogniteExtractorTimeSeries`), and saved in the `extractedData` property. * `data-set-id` configuration option is ignored. ## Using values from Azure Key Vault The PI extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} key-vault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For configuring the earliest point to backfill to you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the backfill only recent history. You can also set the backfill target to a specific date on RFC 3339 form, like `2025-07-02T22:23:12Z`, or on a shorter form like `2022-11-20+12:00`. A timezone specifier is necessary, `Z` means UTC. ## Configure the PI extractor Configuration for the PI extractor. Each section configures a different aspect of the extractor. | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | | [`metrics`](#metrics) | object | Configuration for publishing metrics. | | [`cognite`](#cognite) | object | Configure connection to Cognite Data Fusion (CDF) | | [`state-store`](#state-store) | object | Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | | [`pi`](#pi) | object | Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | | [`time-series`](#time-series) | object | Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. | | [`events`](#events) | object | Configuration for writing events on reconnect and data loss incidents | | [`extractor`](#extractor) | object | The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:

    1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
    2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | | [`backfill`](#backfill) | object | Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | | [`frontfill`](#frontfill) | object | Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. | | [`high-availability`](#high-availability) | object | Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | ##
    `logger` Global parameter. Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | ### `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configuration for publishing metrics. | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. | | [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for having the extractor start a Prometheus scrape server on a local port. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.

    **Examples:**
    `localhost`
    `0.0.0.0` | | `port` | integer | **Required.** The port used for a local Prometheus server. | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://my.pushgateway:9091` | | `job` | string | **Required.** Name of the Prometheus pushgateway job. | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ##
    `cognite` Global parameter. Configure connection to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`metadata-targets`](#cognite.metadata-targets) | object | Configuration for targets for time series metadata. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configuration for targets for time series metadata. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Configuration for writing metadata to CDF Raw. | | [`clean`](#cognite.metadata-targets.clean) | object | Configuration for enabling writing metadata to CDF Clean. | #### `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for writing metadata to CDF Raw. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The Raw database to write to. | | `timeseries-table` | string | **Required.** Name of the Raw table to write timeseries metadata to, enables writing metadata to Raw. Metadata in this case includes name, description, and unit. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for enabling writing metadata to CDF Clean. | Parameter | Type | Description | | - | - | - | | `timeseries` | boolean | Set to `false` to disable writing metadata to time series. Default value is `True`. | ## `state-store` Global parameter. Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Enter the time between each write to the state store. 0 or less disables the state store. Format is as given in [Intervals](#intervals). Default value is `10s`. | | `time-series-table-name` | string | Table name in Raw/Redis/LiteDB for storing time series state. Default value is `ranges`. | | `extractor-table-name` | string | Table name in Raw/Redis/LiteDB for storing general extractor state. Default value is `extractor`. | | `redis` | boolean | Use a redis state store. Overrides `database`. | ## `pi` Global parameter. Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Enter the hostname or IP address of the PI server or the PI collective name. If the host is a collective member name, connect to that member. | | `username` | string | Enter the username to use for authentication. Leave this empty if the Windows service account under which the extractor runs is authorized in Active Directory to access the PI host. | | `password` | string | Enter the password for the given username, if any. | | `native-authentication` | boolean | Determines whether the extractor will use native PI authentication or Windows authentication. The default value is `false`, which indicates Windows authentication. | | `parallelism` | integer | Insert the number of parallel requests to the PI server. If `backfill-parallelism` is set, this excludes backfill requests. Default value is `1`. | | `backfill-parallelism` | integer | Insert the number of parallel requests to the PI server for backfills. This allows the separate throttling of backfills. The default value is 0, which means that the value in `parallelism` is used. | | `use-member-priority` | boolean | When connecting to a PI Collective, attempt to connect to the member with highest priority. If set to `false`, attempt to connect to the member with the same name as `host`. | | `max-connection-retries` | integer | The maximum number of times to attempt to connect to the PI server before failing fatally. If this is `0`, retry forever. | ## `time-series` Global parameter. Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. The example would create time series on the following form in CDF: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi:12345", "name": "PI-Point-Name", "isString": false, "isStep": false, "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} external-id-prefix: 'pi:' external-id-source: SourceId data-set-id: 1234567890123456 ``` | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | Enter the external ID prefix to identify the time series in CDF. Leave empty for no prefix. The external ID in CDF will be this prefix followed by either the PI Point name or PI Point ID. | | `external-id-source` | either `Name` or `SourceId` | Enter the source of the external ID. `Name` means that the PI Point name is used, while `SourceID` means that the PI Point ID is used. Default value is `Name`. | | `data-set-id` | integer | Specify the data set to assign to all time series controlled by the extractor, both new and current. If you don't configure this, the extractor will not change the current time series' data set. | | `data-set-external-id` | string | Specify the external ID of the data set to use, see `data-set-id`. Using this requires the dataSets:READ ACL in CDF | | `sanitation` | either `Remove`, `Clean` or `None` | Specify what to do when the time series fields exceed CDF limits. `Remove` will skip any time series that fail sanitation. `Clean` will truncate and remove values to conform to limits. `None` does nothing (requests may fail as a result). External IDs are never truncated, any time series exceeding CDF limits will be skipped to avoid external ID collisions, regardless of this configuration. Default value is `Clean`. | | `update-metadata` | boolean | Enable updating time series if extractor configuration or PI metadata changes. Default value is `True`. | | `space-id` | string | The identifier for the Data modeling space. This extractor configuration parameter overrides `data-set-id` parameter, and changes the extractor destination from the Asset-Centric TimeSeries to a Core Data Model based TimeSeries, using the `CogniteExtractorTimeSeries` system model extension. | | `source-id` | string | Data modeling source id, this parameter will override the external id of the source node in data modeling, which defaults to the `host` parameter from PI server configuration. | ## `events` Global parameter. Configuration for writing events on reconnect and data loss incidents The example configuration produces events on the following form ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi-events:PiExtractor-2020-08-04 01:01:21.395(0)", "startTime": 1596502850668, "endTime": 1596502880393, "type": "DataLoss", "subtype": "DataPipeOverflow", "source": "PiExtractor", "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} source: PiExtractor external-id-prefix: 'pi-events:' data-set-id: 1234567890123456 store-extractor-events-interval: 5m ``` | Parameter | Type | Description | | - | - | - | | `source` | string | Events have this value as the `source` in CDF. | | `external-id-prefix` | string | Set an external ID prefix to identify events in CDF. Leave empty for no prefix. | | `data-set-id` | integer | Select a data set to assign to all events created by the extractor. We recommend using the same data set ID used in the [Time series](#time-series) section. | | `data-set-external-id` | string | Data set external ID added to events written to CDF, see `data-set-id`. Using this requires the `dataSets:READ` ACL in CDF | | `store-extractor-events-interval` | string | Store events in CDF with this interval. Format is as given in [Intervals](#intervals). If this is not set, events are not created in CDF.

    **Examples:**
    `10s`
    `5m` | ##
    `extractor` Global parameter. The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created: 1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points. 2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | Parameter | Type | Description | | - | - | - | | `attribute-filter-case-insensitive` | boolean | Enable case-insensitive comparison for attribute keys and values in include-attribute-values and exclude-attribute-values filters. When enabled, attribute value 'ABC' will match 'abc', 'Abc', 'ABC', etc. NOTE: This setting does NOT affect tag name matching (include-tags, exclude-tags), prefix matching (include-prefixes, exclude-prefixes), or pattern matching (include-patterns, exclude-patterns), which remain case-sensitive. Default is false (case-sensitive matching). | | [`include-tags`](#extractor.include-tags) | list | Include tags with name exactly equal to one of these names. | | [`include-prefixes`](#extractor.include-prefixes) | list | Include tags with name starting with one of these prefixes. | | [`include-patterns`](#extractor.include-patterns) | list | Include tags with name containing one of these substrings. | | [`include-attribute-values`](#extractor.include-attribute-values) | list | Include tags with attributes equal to at least one of these key/value pairs. | | [`exclude-tags`](#extractor.exclude-tags) | list | Exclude tags with name exactly equal to one of these names. | | [`exclude-prefixes`](#extractor.exclude-prefixes) | list | Exclude tags with name starting with one of these prefixes. | | [`exclude-patterns`](#extractor.exclude-patterns) | list | Exclude tags with name containing one of these substrings. | | [`exclude-attribute-values`](#extractor.exclude-attribute-values) | list | Exclude tags with attributes equal to at least one of these key/value pairs. | | `time-series-update-interval` | either string or integer | Interval between PI tag scans. Each scan discovers new PI tags and detects tags removed from PI but still present in CDF; removed tags are handled per extractor.deleted-time-series.behavior (applies to both classic and CDM time series, plus RAW metadata when enabled). The accepted formats for this field is either in the form of intervals like `4d`, or in the form of a cron expression like `0 4 * * *`. Default value is `24h`. | | [`deleted-time-series`](#extractor.deleted-time-series) | object | Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). | | `end-of-stream-interval` | string | Interval for fetching the end-of-stream timestamps from PI. Format is as given in [Intervals](#intervals). | | `end-of-stream-chunking` | integer | Maximum number of time series per end-of-stream request. Default value is `10000`. | | `end-of-stream-throttle` | integer | Maximum number of parallel end-of-stream requests. Default value is `10`. | | `time-series-update-throttle` | integer | Maximum number of parallel time series updates. Default value is `10`. | | `dry-run` | boolean | Set this to `true` to run the extractor in dry-run mode, where it does not push any data to CDF. Useful for testing the connection to the PI Server | | `read-extracted-ranges` | boolean | Set this to `false` to disable reading from CDF on extractor startup. If this is set to `false`, it is strongly recommended to have a state-store configured, or the extractor will read *all* history from PI on every run. Default value is `True`. | | [`status-codes`](#extractor.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | ### `include-tags` Part of [`extractor`](#extractor) configuration. Include tags with name exactly equal to one of these names. Each element of this list should be a string. ### `include-prefixes` Part of [`extractor`](#extractor) configuration. Include tags with name starting with one of these prefixes. Each element of this list should be a string. ### `include-patterns` Part of [`extractor`](#extractor) configuration. Include tags with name containing one of these substrings. Each element of this list should be a string. ### `include-attribute-values` Part of [`extractor`](#extractor) configuration. Include tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `exclude-tags` Part of [`extractor`](#extractor) configuration. Exclude tags with name exactly equal to one of these names. Each element of this list should be a string. ### `exclude-prefixes` Part of [`extractor`](#extractor) configuration. Exclude tags with name starting with one of these prefixes. Each element of this list should be a string. ### `exclude-patterns` Part of [`extractor`](#extractor) configuration. Exclude tags with name containing one of these substrings. Each element of this list should be a string. ### `exclude-attribute-values` Part of [`extractor`](#extractor) configuration. Exclude tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `deleted-time-series` Part of [`extractor`](#extractor) configuration. Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). This only affects time series with the same data set ID and external ID prefix as the [time series configured in the extractor](#time-series) To find time series that exist in CDF but not in PI, the extractor: * Lists all time series in CDF that have the configured external ID prefix and data set ID * Filters the time series using the include/exclude rules defined in the [extractor](#extractor) section. * Matches the result against the time series obtained from the PI Server after filtering these using the include/exclude rules. | Parameter | Type | Description | | - | - | - | | `behavior` | either `None`, `Flag` or `Delete` | Select the action taken by the extractor. Setting this to `Flag` will perform soft deletion: Flag the time series as deleted but don't delete them from CDF. Setting it to `Delete` will delete the time series from CDF.

    **If you set this to `Delete`, the time series in CDF that cannot be found in PI will be permanently deleted from CDF.**. Default value is `None`. | | `flag-name` | string | If you've set `behavior` to `Flag`, use this string to mark the time series as deleted. Metadata is added to the time series with this as the key, and the current time as the value. Default value is `deletedByExtractor`. | | `time-series-delete-throttle` | integer | Maximum number of parallel delete operations. Default value is `10`. | ###
    `status-codes` Part of [`extractor`](#extractor) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | ## `backfill` Global parameter. Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set to `true` to disable backfill. | | `step-size-hours` | integer | Step, in whole number of hours. Set to 0 to freely backfill all time series. Each iteration of backfill will backfill all time series to the next step before stepping further backward. This helps even out the progression of long backfill processes. Default value is `168`. | | `to` | string | The target CDF timestamp in miliseconds at which to stop the backfill. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Examples:**
    `3d-ago`
    `2025-07-02T22:23:12Z`
    `2022-11-20+12:00` | | `time-series-delay` | integer | Delay in milliseconds between each time series backfill request within a step. | | `step-delay` | integer | Delay in milliseconds between each step. | | `retry-bucket-size` | integer | Maximum size of the retry bucket. Zero or less for no limit. Default value is `100`. | | `retry-bucket-cleanup-time` | integer | Delay in seconds between each cleanup of the retry bucket. Default value is `3600`. | ##
    `frontfill` Global parameter. Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. When data points are archived in PI, they may be subject to compression, reducing the total amount of data points in a time series. Therefore, the backfill and frontfill tasks will receive data points after compression, while the streaming task will receive data points before compression. Learn more about compression in [this video](https://www.youtube.com/watch?v=6-scv3oQ7Kk). | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set this to `true` to disable frontfill and streaming. | | `streaming-interval` | string | Interval between each call to the PI Data Pipe to fetch new data. Format is as given in [Intervals](#intervals). If set to zero, the extractor will not fetch live data from PI.

    **If you set this parameter to a high value, there is a higher chance of having a client outbound queue overflow in the PI server. Overflow may cause loss of data points in CDF.**

    Recommended range is between 1 and 30 seconds. If you observe data loss events, consider turning this value down. Default value is `1s`. | | `delete-data-points` | boolean | If you set this to `true`, the corresponding data points are deleted in CDF when data point deletion events are received in the PI Data Pipe.

    **Enabling this parameter may increase the streaming latency of the extractor since the extractor verifies the data point deletion before proceeding with the insertions.** | | `use-data-pipe` | boolean | Older PI servers may not support data pipes. If that's the case, set this value to `false` to disable data pipe streaming. The frontfiller task will run constantly and will frequently query the PI Data Archive for new data points. Default value is `True`. | | `time-series-chunk` | integer | The maximum number of time series in each frontfill query to PI. Value of this parameter cannot be larger than the value of `data-points-chunk`. Default value is `1000`. | | `data-points-chunk` | integer | The maximum number of requested data points in each frontfill query to PI. Default value is `10000`. | ##
    `high-availability` Global parameter. Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | Parameter | Type | Description | | - | - | - | | `interval` | string | Interval to update the high availability state in Redis. Format is as given in [Intervals](#intervals). | | `timeout` | integer | Timeout in seconds before taking over as active extractor. Must be set greater than 0 to enable high availability. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/4-22 PI extractor configuration reference for version 4.22. This page documents PI extractor **4.22**. For the current release, see [Configuration settings](/cdf/integration/guides/extraction/pi/pi). To configure the PI extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org/) format. The configuration file is split into [sections](#configure-the-pi-extractor), each represented by a top-level entry in the YAML format. Subsections are nested under a section in the YAML format. You can use either the sample complete or minimal configuration files included with the [installer](../pi/pi_setup.md) as a starting point for your configuration settings: * config.default.yml - This file contains all configuration options and descriptions. * config.minimal.yml - This file contains a minimum configuration and no descriptions. You must name the configuration file *config.yml*. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Before you start * Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`. * The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway. * Set up an [extraction pipeline](../../interfaces/add_integrations.md) and note the external ID. ## Minimal YAML configuration file The YAML settings below contain valid PI extractor 2.1 configurations. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file has a global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 3 of the configuration schema. ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 3 cognite: project: '${COGNITE_PROJECT}' idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} time-series: external-id-prefix: 'pi:' pi: host: ${PI_HOST} username: ${PI_USER} password: ${PI_PASSWORD} state-store: database: LiteDb location: 'state.db' logger: file: level: 'information' path: 'logs/log.log' ``` where: * `version` is the version of the configuration schema. Use version 3 to be compatible with the Cognite PI extractor 2.1. * `cognite` is the how the extractor reads the authentication details for PI (pi) and CDF (cognite) from environment variables. Since no host is specified in the `cognite` section, the extractor uses the default value, \<[https://api.cognitedata.com](https://api.cognitedata.com) >, and assumes that the PI server uses Windows authentication. * `time-series` configures the extractor to create time series in CDF where the external IDs will be prefixed with `pi:`. You can also use a data set ID configuration to add all time series created by the extractor to a particular data set. * `state-store` configures the extractor to save the extraction state locally using a LiteDB database file named `state.db`. * `logger` configures the extractor to log at information level and outputs log messages to a log file in the logs/log.log directory. By default, new files are created daily and retained for 31 days. The date is appended to the file name. ## Writing to CDF Data Models The extractor can write directly to a [core data model time series](../../../../dm/dm_reference/dm_core_data_model#timeseries), or more correctly, to an extended version of the `CogniteTimeSeries` Data Model type. To save/update instances of `CogniteTimeSeries`, specify the [identifier for the target Data Modeling Space](https://api-docs.cognite.com/20230101/tag/Spaces/operation/ApplySpaces#!path=items/space\&t=request) in the configuration, using the `space-id` parameter of the [time series configuration](#-time-series) section. The behavior in this mode is almost identical to classic time series destination, except for a few key differences: * Metadata is written as a json blob to the extended time series type (`CogniteExtractorTimeSeries`), and saved in the `extractedData` property. * `data-set-id` configuration option is ignored. ## Using values from Azure Key Vault The PI extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} key-vault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For configuring the earliest point to backfill to you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the backfill only recent history. You can also set the backfill target to a specific date on RFC 3339 form, like `2025-07-02T22:23:12Z`, or on a shorter form like `2022-11-20+12:00`. A timezone specifier is necessary, `Z` means UTC. ## Configure the PI extractor Configuration for the PI extractor. Each section configures a different aspect of the extractor. | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | | [`metrics`](#metrics) | object | Configuration for publishing metrics. | | [`cognite`](#cognite) | object | Configure connection to Cognite Data Fusion (CDF) | | [`state-store`](#state-store) | object | Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | | [`pi`](#pi) | object | Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | | [`time-series`](#time-series) | object | Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. | | [`events`](#events) | object | Configuration for writing events on reconnect and data loss incidents | | [`extractor`](#extractor) | object | The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:

    1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
    2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | | [`backfill`](#backfill) | object | Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | | [`frontfill`](#frontfill) | object | Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. | | [`high-availability`](#high-availability) | object | Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | ##
    `logger` Global parameter. Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | ### `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configuration for publishing metrics. | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. | | [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for having the extractor start a Prometheus scrape server on a local port. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.

    **Examples:**
    `localhost`
    `0.0.0.0` | | `port` | integer | **Required.** The port used for a local Prometheus server. | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://my.pushgateway:9091` | | `job` | string | **Required.** Name of the Prometheus pushgateway job. | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ##
    `cognite` Global parameter. Configure connection to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`metadata-targets`](#cognite.metadata-targets) | object | Configuration for targets for time series metadata. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configuration for targets for time series metadata. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Configuration for writing metadata to CDF Raw. | | [`clean`](#cognite.metadata-targets.clean) | object | Configuration for enabling writing metadata to CDF Clean. | #### `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for writing metadata to CDF Raw. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The Raw database to write to. | | `timeseries-table` | string | **Required.** Name of the Raw table to write timeseries metadata to, enables writing metadata to Raw. Metadata in this case includes name, description, and unit. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for enabling writing metadata to CDF Clean. | Parameter | Type | Description | | - | - | - | | `timeseries` | boolean | Set to `false` to disable writing metadata to time series. Default value is `True`. | ## `state-store` Global parameter. Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Enter the time between each write to the state store. 0 or less disables the state store. Format is as given in [Intervals](#intervals). Default value is `10s`. | | `time-series-table-name` | string | Table name in Raw/Redis/LiteDB for storing time series state. Default value is `ranges`. | | `extractor-table-name` | string | Table name in Raw/Redis/LiteDB for storing general extractor state. Default value is `extractor`. | | `redis` | boolean | Use a redis state store. Overrides `database`. | ## `pi` Global parameter. Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Enter the hostname or IP address of the PI server or the PI collective name. If the host is a collective member name, connect to that member. | | `username` | string | Enter the username to use for authentication. Leave this empty if the Windows service account under which the extractor runs is authorized in Active Directory to access the PI host. | | `password` | string | Enter the password for the given username, if any. | | `native-authentication` | boolean | Determines whether the extractor will use native PI authentication or Windows authentication. The default value is `false`, which indicates Windows authentication. | | `parallelism` | integer | Insert the number of parallel requests to the PI server. If `backfill-parallelism` is set, this excludes backfill requests. Default value is `1`. | | `backfill-parallelism` | integer | Insert the number of parallel requests to the PI server for backfills. This allows the separate throttling of backfills. The default value is 0, which means that the value in `parallelism` is used. | | `use-member-priority` | boolean | When connecting to a PI Collective, attempt to connect to the member with highest priority. If set to `false`, attempt to connect to the member with the same name as `host`. | | `max-connection-retries` | integer | The maximum number of times to attempt to connect to the PI server before failing fatally. If this is `0`, retry forever. | ## `time-series` Global parameter. Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. The example would create time series on the following form in CDF: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi:12345", "name": "PI-Point-Name", "isString": false, "isStep": false, "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} external-id-prefix: 'pi:' external-id-source: SourceId data-set-id: 1234567890123456 ``` | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | Enter the external ID prefix to identify the time series in CDF. Leave empty for no prefix. The external ID in CDF will be this prefix followed by either the PI Point name or PI Point ID. | | `external-id-source` | either `Name` or `SourceId` | Enter the source of the external ID. `Name` means that the PI Point name is used, while `SourceID` means that the PI Point ID is used. Default value is `Name`. | | `data-set-id` | integer | Specify the data set to assign to all time series controlled by the extractor, both new and current. If you don't configure this, the extractor will not change the current time series' data set. | | `data-set-external-id` | string | Specify the external ID of the data set to use, see `data-set-id`. Using this requires the dataSets:READ ACL in CDF | | `sanitation` | either `Remove`, `Clean` or `None` | Specify what to do when the time series fields exceed CDF limits. `Remove` will skip any time series that fail sanitation. `Clean` will truncate and remove values to conform to limits. `None` does nothing (requests may fail as a result). External IDs are never truncated, any time series exceeding CDF limits will be skipped to avoid external ID collisions, regardless of this configuration. Default value is `Clean`. | | `update-metadata` | boolean | Enable updating time series if extractor configuration or PI metadata changes. Default value is `True`. | | `space-id` | string | The identifier for the Data modeling space. This extractor configuration parameter overrides `data-set-id` parameter, and changes the extractor destination from the Asset-Centric TimeSeries to a Core Data Model based TimeSeries, using the `CogniteExtractorTimeSeries` system model extension. | | `source-id` | string | Data modeling source id, this parameter will override the external id of the source node in data modeling, which defaults to the `host` parameter from PI server configuration. | ## `events` Global parameter. Configuration for writing events on reconnect and data loss incidents The example configuration produces events on the following form ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi-events:PiExtractor-2020-08-04 01:01:21.395(0)", "startTime": 1596502850668, "endTime": 1596502880393, "type": "DataLoss", "subtype": "DataPipeOverflow", "source": "PiExtractor", "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} source: PiExtractor external-id-prefix: 'pi-events:' data-set-id: 1234567890123456 store-extractor-events-interval: 5m ``` | Parameter | Type | Description | | - | - | - | | `source` | string | Events have this value as the `source` in CDF. | | `external-id-prefix` | string | Set an external ID prefix to identify events in CDF. Leave empty for no prefix. | | `data-set-id` | integer | Select a data set to assign to all events created by the extractor. We recommend using the same data set ID used in the [Time series](#time-series) section. | | `data-set-external-id` | string | Data set external ID added to events written to CDF, see `data-set-id`. Using this requires the `dataSets:READ` ACL in CDF | | `store-extractor-events-interval` | string | Store events in CDF with this interval. Format is as given in [Intervals](#intervals). If this is not set, events are not created in CDF.

    **Examples:**
    `10s`
    `5m` | ##
    `extractor` Global parameter. The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created: 1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points. 2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | Parameter | Type | Description | | - | - | - | | `attribute-filter-case-insensitive` | boolean | Enable case-insensitive comparison for attribute keys and values in include-attribute-values and exclude-attribute-values filters. When enabled, attribute value 'ABC' will match 'abc', 'Abc', 'ABC', etc. NOTE: This setting does NOT affect tag name matching (include-tags, exclude-tags), prefix matching (include-prefixes, exclude-prefixes), or pattern matching (include-patterns, exclude-patterns), which remain case-sensitive. Default is false (case-sensitive matching). | | [`include-tags`](#extractor.include-tags) | list | Include tags with name exactly equal to one of these names. | | [`include-prefixes`](#extractor.include-prefixes) | list | Include tags with name starting with one of these prefixes. | | [`include-patterns`](#extractor.include-patterns) | list | Include tags with name containing one of these substrings. | | [`include-attribute-values`](#extractor.include-attribute-values) | list | Include tags with attributes equal to at least one of these key/value pairs. | | [`exclude-tags`](#extractor.exclude-tags) | list | Exclude tags with name exactly equal to one of these names. | | [`exclude-prefixes`](#extractor.exclude-prefixes) | list | Exclude tags with name starting with one of these prefixes. | | [`exclude-patterns`](#extractor.exclude-patterns) | list | Exclude tags with name containing one of these substrings. | | [`exclude-attribute-values`](#extractor.exclude-attribute-values) | list | Exclude tags with attributes equal to at least one of these key/value pairs. | | `time-series-update-interval` | either string or integer | Interval between PI tag scans. Each scan discovers new PI tags and detects tags removed from PI but still present in CDF; removed tags are handled per extractor.deleted-time-series.behavior (applies to both classic and CDM time series, plus RAW metadata when enabled). The accepted formats for this field is either in the form of intervals like `4d`, or in the form of a cron expression like `0 4 * * *`. Default value is `24h`. | | [`deleted-time-series`](#extractor.deleted-time-series) | object | Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). | | `end-of-stream-interval` | string | Interval for fetching the end-of-stream timestamps from PI. Format is as given in [Intervals](#intervals). | | `end-of-stream-chunking` | integer | Maximum number of time series per end-of-stream request. Default value is `10000`. | | `end-of-stream-throttle` | integer | Maximum number of parallel end-of-stream requests. Default value is `10`. | | `time-series-update-throttle` | integer | Maximum number of parallel time series updates. Default value is `10`. | | `dry-run` | boolean | Set this to `true` to run the extractor in dry-run mode, where it does not push any data to CDF. Useful for testing the connection to the PI Server | | `read-extracted-ranges` | boolean | Set this to `false` to disable reading from CDF on extractor startup. If this is set to `false`, it is strongly recommended to have a state-store configured, or the extractor will read *all* history from PI on every run. Default value is `True`. | | [`status-codes`](#extractor.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | ### `include-tags` Part of [`extractor`](#extractor) configuration. Include tags with name exactly equal to one of these names. Each element of this list should be a string. ### `include-prefixes` Part of [`extractor`](#extractor) configuration. Include tags with name starting with one of these prefixes. Each element of this list should be a string. ### `include-patterns` Part of [`extractor`](#extractor) configuration. Include tags with name containing one of these substrings. Each element of this list should be a string. ### `include-attribute-values` Part of [`extractor`](#extractor) configuration. Include tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `exclude-tags` Part of [`extractor`](#extractor) configuration. Exclude tags with name exactly equal to one of these names. Each element of this list should be a string. ### `exclude-prefixes` Part of [`extractor`](#extractor) configuration. Exclude tags with name starting with one of these prefixes. Each element of this list should be a string. ### `exclude-patterns` Part of [`extractor`](#extractor) configuration. Exclude tags with name containing one of these substrings. Each element of this list should be a string. ### `exclude-attribute-values` Part of [`extractor`](#extractor) configuration. Exclude tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `deleted-time-series` Part of [`extractor`](#extractor) configuration. Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). This only affects time series with the same data set ID and external ID prefix as the [time series configured in the extractor](#time-series) To find time series that exist in CDF but not in PI, the extractor: * Lists all time series in CDF that have the configured external ID prefix and data set ID * Filters the time series using the include/exclude rules defined in the [extractor](#extractor) section. * Matches the result against the time series obtained from the PI Server after filtering these using the include/exclude rules. | Parameter | Type | Description | | - | - | - | | `behavior` | either `None`, `Flag` or `Delete` | Select the action taken by the extractor. Setting this to `Flag` will perform soft deletion: Flag the time series as deleted but don't delete them from CDF. Setting it to `Delete` will delete the time series from CDF.

    **If you set this to `Delete`, the time series in CDF that cannot be found in PI will be permanently deleted from CDF.**. Default value is `None`. | | `flag-name` | string | If you've set `behavior` to `Flag`, use this string to mark the time series as deleted. Metadata is added to the time series with this as the key, and the current time as the value. Default value is `deletedByExtractor`. | | `time-series-delete-throttle` | integer | Maximum number of parallel delete operations. Default value is `10`. | ###
    `status-codes` Part of [`extractor`](#extractor) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | ## `backfill` Global parameter. Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set to `true` to disable backfill. | | `step-size-hours` | integer | Step, in whole number of hours. Set to 0 to freely backfill all time series. Each iteration of backfill will backfill all time series to the next step before stepping further backward. This helps even out the progression of long backfill processes. Default value is `168`. | | `to` | string | The target CDF timestamp in miliseconds at which to stop the backfill. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Examples:**
    `3d-ago`
    `2025-07-02T22:23:12Z`
    `2022-11-20+12:00` | | `time-series-delay` | integer | Delay in milliseconds between each time series backfill request within a step. | | `step-delay` | integer | Delay in milliseconds between each step. | | `retry-bucket-size` | integer | Maximum size of the retry bucket. Zero or less for no limit. Default value is `100`. | | `retry-bucket-cleanup-time` | integer | Delay in seconds between each cleanup of the retry bucket. Default value is `3600`. | ##
    `frontfill` Global parameter. Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. When data points are archived in PI, they may be subject to compression, reducing the total amount of data points in a time series. Therefore, the backfill and frontfill tasks will receive data points after compression, while the streaming task will receive data points before compression. Learn more about compression in [this video](https://www.youtube.com/watch?v=6-scv3oQ7Kk). | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set this to `true` to disable frontfill and streaming. | | `streaming-interval` | string | Interval between each call to the PI Data Pipe to fetch new data. Format is as given in [Intervals](#intervals). If set to zero, the extractor will not fetch live data from PI.

    **If you set this parameter to a high value, there is a higher chance of having a client outbound queue overflow in the PI server. Overflow may cause loss of data points in CDF.**

    Recommended range is between 1 and 30 seconds. If you observe data loss events, consider turning this value down. Default value is `1s`. | | `delete-data-points` | boolean | If you set this to `true`, the corresponding data points are deleted in CDF when data point deletion events are received in the PI Data Pipe.

    **Enabling this parameter may increase the streaming latency of the extractor since the extractor verifies the data point deletion before proceeding with the insertions.** | | `use-data-pipe` | boolean | Older PI servers may not support data pipes. If that's the case, set this value to `false` to disable data pipe streaming. The frontfiller task will run constantly and will frequently query the PI Data Archive for new data points. Default value is `True`. | | `time-series-chunk` | integer | The maximum number of time series in each frontfill query to PI. Value of this parameter cannot be larger than the value of `data-points-chunk`. Default value is `1000`. | | `data-points-chunk` | integer | The maximum number of requested data points in each frontfill query to PI. Default value is `10000`. | | `treat-missing-as-no-data` | boolean | If true, when PI returns no data points for a tag during frontfill, the extractor advances that tag's cursor to the current time instead of leaving it unmoved. Only applicable when use-data-pipe is false, will be a no-op when use-data-pipe is true.

    WARNING: enabling this will skip any historical data that arrives after the cursor has been advanced. Only enable this if you are certain the historical data won't arrive later. | ##
    `high-availability` Global parameter. Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | Parameter | Type | Description | | - | - | - | | `interval` | string | Interval to update the high availability state in Redis. Format is as given in [Intervals](#intervals). | | `timeout` | integer | Timeout in seconds before taking over as active extractor. Must be set greater than 0 to enable high availability. | # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/pi Complete reference for configuring the Cognite PI extractor version 4.23. This page documents PI extractor **4.23** (current release). For older releases, see **Other versions** in the sidebar. To configure the PI extractor, you must create a configuration file. This file must be in [YAML](https://yaml.org/) format. The configuration file is split into [sections](#configure-the-pi-extractor), each represented by a top-level entry in the YAML format. Subsections are nested under a section in the YAML format. You can use either the sample complete or minimal configuration files included with the [installer](../pi/pi_setup.md) as a starting point for your configuration settings: * config.default.yml - This file contains all configuration options and descriptions. * config.minimal.yml - This file contains a minimum configuration and no descriptions. You must name the configuration file *config.yml*. You can set up [extraction pipelines](../../interfaces/configure_integrations.md) to use versioned extractor configuration files stored in the cloud. ## Before you start * Optionally, copy one of the sample files in the `config` directory and rename it to `config.yml`. * The `config.minimal.yml` file doesn't include a metrics section. Copy this section from the example below if the extractor is required to send metrics to a Prometheus Pushgateway. * Set up an [extraction pipeline](../../interfaces/add_integrations.md) and note the external ID. ## Minimal YAML configuration file The YAML settings below contain valid PI extractor 2.1 configurations. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file has a global parameter version, which holds the version of the configuration schema used in the configuration file. This document describes version 3 of the configuration schema. ```yml showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 3 cognite: project: '${COGNITE_PROJECT}' idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} time-series: external-id-prefix: 'pi:' pi: host: ${PI_HOST} username: ${PI_USER} password: ${PI_PASSWORD} state-store: database: LiteDb location: 'state.db' logger: file: level: 'information' path: 'logs/log.log' ``` where: * `version` is the version of the configuration schema. Use version 3 to be compatible with the Cognite PI extractor 2.1. * `cognite` is the how the extractor reads the authentication details for PI (pi) and CDF (cognite) from environment variables. Since no host is specified in the `cognite` section, the extractor uses the default value, \<[https://api.cognitedata.com](https://api.cognitedata.com) >, and assumes that the PI server uses Windows authentication. * `time-series` configures the extractor to create time series in CDF where the external IDs will be prefixed with `pi:`. You can also use a data set ID configuration to add all time series created by the extractor to a particular data set. * `state-store` configures the extractor to save the extraction state locally using a LiteDB database file named `state.db`. * `logger` configures the extractor to log at information level and outputs log messages to a log file in the logs/log.log directory. By default, new files are created daily and retained for 31 days. The date is appended to the file name. ## Writing to CDF Data Models The extractor can write directly to a [core data model time series](../../../../dm/dm_reference/dm_core_data_model#timeseries), or more correctly, to an extended version of the `CogniteTimeSeries` Data Model type. To save/update instances of `CogniteTimeSeries`, specify the [identifier for the target Data Modeling Space](https://api-docs.cognite.com/20230101/tag/Spaces/operation/ApplySpaces#!path=items/space\&t=request) in the configuration, using the `space-id` parameter of the [time series configuration](#-time-series) section. The behavior in this mode is almost identical to classic time series destination, except for a few key differences: * Metadata is written as a json blob to the extended time series type (`CogniteExtractorTimeSeries`), and saved in the `extractedData` property. * `data-set-id` configuration option is ignored. ## Using values from Azure Key Vault The PI extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `key-vault` section in your configuration, with the following parameters: | Parameter | Description | | - | - | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} key-vault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` ## Timestamps and intervals In most places where time intervals are required, you can use a CDF-like syntax of `[N][timeunit]`, for example `10m` for 10 minutes or `1h` for 1 hour. `timeunit` is one of `d`, `h`, `m`, `s`, `ms`. You can also use a cron expression in some places. For configuring the earliest point to backfill to you can use a similar syntax. `[N][timeunit]` and `[N][timeunit]-ago`. `1d-ago` means 1 day in the past from the time history starts, and `1h` means 1 hour in the future. For instance, you can use this syntax to configure the backfill only recent history. You can also set the backfill target to a specific date on RFC 3339 form, like `2025-07-02T22:23:12Z`, or on a shorter form like `2022-11-20+12:00`. A timezone specifier is necessary, `Z` means UTC. ## Configure the PI extractor Configuration for the PI extractor. Each section configures a different aspect of the extractor. | Parameter | Type | Description | | - | - | - | | `version` | integer | Version of the config file, the extractor specifies which config file versions are accepted in each version of the extractor. | | [`logger`](#logger) | object | Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | | [`metrics`](#metrics) | object | Configuration for publishing metrics. | | [`cognite`](#cognite) | object | Configure connection to Cognite Data Fusion (CDF) | | [`state-store`](#state-store) | object | Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | | [`pi`](#pi) | object | Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | | [`time-series`](#time-series) | object | Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. | | [`events`](#events) | object | Configuration for writing events on reconnect and data loss incidents | | [`extractor`](#extractor) | object | The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created:

    1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points.
    2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | | [`backfill`](#backfill) | object | Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | | [`frontfill`](#frontfill) | object | Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. | | [`high-availability`](#high-availability) | object | Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | ##
    `logger` Global parameter. Configuration for logging to console or file. Log entries are either `Fatal`, `Error`, `Warning`, `Information`, `Debug`, or `Verbose`, in order of decreasing priority. The extractor will log any messages at an equal or higher log level than the configured level for each sink. | Parameter | Type | Description | | - | - | - | | [`console`](#logger.console) | object | Configuration for logging to the console. | | [`file`](#logger.file) | object | Configuration for logging to a rotating log file. | | [`trace-listener`](#logger.trace-listener) | object | Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | ### `console` Part of [`logger`](#logger) configuration. Configuration for logging to the console. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to the console. If not present, or invalid, logging to console is disabled. | | `stderr-level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | Log events at this level or above are redirected to standard error. | ### `file` Part of [`logger`](#logger) configuration. Configuration for logging to a rotating log file. | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Minimum level of log events to write to file. | | `path` | string | **Required.** Path to the files to be logged. If this is set to `logs/log.txt`, logs on the form `logs/log[date].txt` will be created, depending on `rolling-interval`. | | `retention-limit` | integer | Maximum number of log files that are kept in the log folder. Default value is `31`. | | `rolling-interval` | either `day` or `hour` | Rolling interval for log files. Default value is `day`. | ### `trace-listener` Part of [`logger`](#logger) configuration. Adds a listener that uses the configured logger to output messages from `System.Diagnostics.Trace` | Parameter | Type | Description | | - | - | - | | `level` | either `verbose`, `debug`, `information`, `warning`, `error` or `fatal` | **Required.** Level to output trace messages at | ## `metrics` Global parameter. Configuration for publishing metrics. | Parameter | Type | Description | | - | - | - | | [`server`](#metrics.server) | object | Configuration for having the extractor start a Prometheus scrape server on a local port. | | [`push-gateways`](#metrics.push-gateways) | list | A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | ### `server` Part of [`metrics`](#metrics) configuration. Configuration for having the extractor start a Prometheus scrape server on a local port. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Host name for local Prometheus server, must be exposed to some prometheus instance for scraping.

    **Examples:**
    `localhost`
    `0.0.0.0` | | `port` | integer | **Required.** The port used for a local Prometheus server. | ###
    `push-gateways` Part of [`metrics`](#metrics) configuration. A list of pushgateway destinations to push metrics to. The extractor will automatically push metrics to each of these. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URI of the pushgateway host

    **Example:**
    `http://my.pushgateway:9091` | | `job` | string | **Required.** Name of the Prometheus pushgateway job. | | `username` | string | Username for basic authentication | | `password` | string | Password for basic authentication | | `push-interval` | integer | Interval in seconds between each push to the gateway. Default value is `1`. | ##
    `cognite` Global parameter. Configure connection to Cognite Data Fusion (CDF) | Parameter | Type | Description | | - | - | - | | `project` | string | CDF project to connect to. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
    See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | [`cdf-retries`](#cognite.cdf-retries) | object | Configure automatic retries on requests to CDF. | | [`cdf-chunking`](#cognite.cdf-chunking) | object | Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | | [`cdf-throttling`](#cognite.cdf-throttling) | object | Configure the maximum number of parallel requests for different CDF resources. | | [`sdk-logging`](#cognite.sdk-logging) | object | Configure logging of requests from the SDK | | `nan-replacement` | either number or null | Replacement for NaN values when writing to CDF. If left out, NaN values are skipped. | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Configure an associated extraction pipeline | | [`certificates`](#cognite.certificates) | object | Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | | [`metadata-targets`](#cognite.metadata-targets) | object | Configuration for targets for time series metadata. | ###
    `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). See [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) | Parameter | Type | Description | | - | - | - | | `authority` | string | AInsert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | `scopes` | configuration for either list or string | | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | - | - | - | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `cdf-retries` Part of [`cognite`](#cognite) configuration. Configure automatic retries on requests to CDF. | Parameter | Type | Description | | - | - | - | | `timeout` | integer | Timeout in milliseconds for each individual request to CDF. Default value is `80000`. | | `max-retries` | integer | Maximum number of retries on requests to CDF. If this is less than 0, retry forever. Default value is `5`. | | `max-delay` | integer | Max delay in milliseconds between each retry. Base delay is calculated according to 125\*2^retry milliseconds. If less than 0, there is no maximum. Default value is `5000`. | ### `cdf-chunking` Part of [`cognite`](#cognite) configuration. Configure chunking of data on requests to CDF. Note that increasing these may cause requests to fail due to limits in the API itself | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of timeseries per get/create timeseries request. Default value is `1000`. | | `assets` | integer | Maximum number of assets per get/create assets request. Default value is `1000`. | | `data-point-time-series` | integer | Maximum number of timeseries per datapoint create request. Default value is `10000`. | | `data-point-delete` | integer | Maximum number of ranges per delete datapoints request. Default value is `10000`. | | `data-point-list` | integer | Maximum number of timeseries per datapoint read request. Used when getting the first point in a timeseries. Default value is `100`. | | `data-points` | integer | Maximum number of datapoints per datapoints create request. Default value is `100000`. | | `data-points-gzip-limit` | integer | Minimum number of datapoints in request to switch to using gzip. Set to -1 to disable, and 0 to always enable (not recommended). The minimum HTTP packet size is generally 1500 bytes, so this should never be set below 100 for numeric datapoints. Even for larger packages gzip is efficient enough that packages are compressed below 1500 bytes. At 5000 it is always a performance gain. It can be set lower if bandwidth is a major issue. Default value is `5000`. | | `raw-rows` | integer | Maximum number of rows per request to cdf raw. Default value is `10000`. | | `raw-rows-delete` | integer | Maximum number of row keys per delete request to raw. Default value is `1000`. | | `data-point-latest` | integer | Maximum number of timeseries per datapoint read latest request. Default value is `100`. | | `events` | integer | Maximum number of events per get/create events request. Default value is `1000`. | | `sequences` | integer | Maximum number of sequences per get/create sequences request. Default value is `1000`. | | `sequence-row-sequences` | integer | Maximum number of sequences per create sequence rows request. Default value is `1000`. | | `sequence-rows` | integer | Maximum number of sequence rows per sequence when creating rows. Default value is `10000`. | | `instances` | integer | Maximum number of data modeling instances per get/create instance request. Default value is `1000`. | ### `cdf-throttling` Part of [`cognite`](#cognite) configuration. Configure the maximum number of parallel requests for different CDF resources. | Parameter | Type | Description | | - | - | - | | `time-series` | integer | Maximum number of parallel requests per timeseries operation. Default value is `20`. | | `assets` | integer | Maximum number of parallel requests per assets operation. Default value is `20`. | | `data-points` | integer | Maximum number of parallel requests per datapoints operation. Default value is `10`. | | `raw` | integer | Maximum number of parallel requests per raw operation. Default value is `10`. | | `ranges` | integer | Maximum number of parallel requests per get first/last datapoint operation. Default value is `20`. | | `events` | integer | Maximum number of parallel requests per events operation. Default value is `20`. | | `sequences` | integer | Maximum number of parallel requests per sequences operation. Default value is `10`. | | `instances` | integer | Maximum number of parallel requests per data modeling instances operation. Default value is `4`. | ### `sdk-logging` Part of [`cognite`](#cognite) configuration. Configure logging of requests from the SDK | Parameter | Type | Description | | - | - | - | | `disable` | boolean | True to disable logging from the SDK, it is enabled by default | | `level` | either `trace`, `debug`, `information`, `warning`, `error`, `critical` or `none` | Log level to log messages from the SDK at. Default value is `debug`. | | `format` | string | Format of the log message. Default value is `CDF ({Message}): {HttpMethod} {Url} {ResponseHeader[X-Request-ID]} - {Elapsed} ms`. | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Configure an associated extraction pipeline | Parameter | Type | Description | | - | - | - | | `external-id` | string | External ID of the extraction pipeline | | `frequency` | integer | Frequency to report `Seen` to the extraction pipeline in seconds. Less than or equal to zero will not report automatically. Default value is `600`. | ### `certificates` Part of [`cognite`](#cognite) configuration. Configure special handling of SSL certificates. This should never be considered a permanent solution to certificate problems | Parameter | Type | Description | | - | - | - | | `accept-all` | boolean | Accept all remote SSL certificates. This introduces a severe risk of man-in-the-middle attacks | | [`allow-list`](#cognite.certificates.allow-list) | list | List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates | #### `allow-list` Part of [`certificates`](#cognite.certificates) configuration. List of certificate thumbprints to automatically accept. This is a much smaller risk than accepting all certificates Each element of this list should be a string. ### `metadata-targets` Part of [`cognite`](#cognite) configuration. Configuration for targets for time series metadata. | Parameter | Type | Description | | - | - | - | | [`raw`](#cognite.metadata-targets.raw) | object | Configuration for writing metadata to CDF Raw. | | [`clean`](#cognite.metadata-targets.clean) | object | Configuration for enabling writing metadata to CDF Clean. | #### `raw` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for writing metadata to CDF Raw. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The Raw database to write to. | | `timeseries-table` | string | **Required.** Name of the Raw table to write timeseries metadata to, enables writing metadata to Raw. Metadata in this case includes name, description, and unit. | #### `clean` Part of [`metadata-targets`](#cognite.metadata-targets) configuration. Configuration for enabling writing metadata to CDF Clean. | Parameter | Type | Description | | - | - | - | | `timeseries` | boolean | Set to `false` to disable writing metadata to time series. Default value is `True`. | ## `state-store` Global parameter. Include the `state-store` section to configure the extractor to save the extraction state periodically. This makes the extraction resume faster in the next run. This section is optional. If not present, or if `database` is set to `none`, the extraction state is restored by querying the timestamps of the first and last data points of each time series. If CDF Raw is used as a state store, you can see the extracted ranges under **Manage staged data** in CDF. | Parameter | Type | Description | | - | - | - | | `location` | string | **Required.** Path to .db file used for storage, or name of a CDF RAW database. | | `database` | either `None`, `LiteDb` or `Raw` | Which type of database to use. Default value is `None`. | | `interval` | string | Enter the time between each write to the state store. 0 or less disables the state store. Format is as given in [Intervals](#intervals). Default value is `10s`. | | `time-series-table-name` | string | Table name in Raw/Redis/LiteDB for storing time series state. Default value is `ranges`. | | `extractor-table-name` | string | Table name in Raw/Redis/LiteDB for storing general extractor state. Default value is `extractor`. | | `redis` | boolean | Use a redis state store. Overrides `database`. | ## `pi` Global parameter. Configure the extractor to connect to a particular PI server or PI collective. If you configure the extractor with a PI collective, the extractor will transparently maintain a connection to one of the active servers in the collective. The default settings provide Active Directory authorization to the PI host when the server account for the Windows service is authorized. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** Enter the hostname or IP address of the PI server or the PI collective name. If the host is a collective member name, connect to that member. | | `username` | string | Enter the username to use for authentication. Leave this empty if the Windows service account under which the extractor runs is authorized in Active Directory to access the PI host. | | `password` | string | Enter the password for the given username, if any. | | `native-authentication` | boolean | Determines whether the extractor will use native PI authentication or Windows authentication. The default value is `false`, which indicates Windows authentication. | | `parallelism` | integer | Insert the number of parallel requests to the PI server. If `backfill-parallelism` is set, this excludes backfill requests. Default value is `1`. | | `backfill-parallelism` | integer | Insert the number of parallel requests to the PI server for backfills. This allows the separate throttling of backfills. The default value is 0, which means that the value in `parallelism` is used. | | `use-member-priority` | boolean | When connecting to a PI Collective, attempt to connect to the member with highest priority. If set to `false`, attempt to connect to the member with the same name as `host`. | | `max-connection-retries` | integer | The maximum number of times to attempt to connect to the PI server before failing fatally. If this is `0`, retry forever. | | [`excluded-attributes`](#pi.excluded-attributes) | list | PI point attributes to exclude from being read into time series metadata. If left unset (default), a built-in list of internal PI attributes is excluded (`creationdate`, `creator`, `changedate`, `changer`, `ptowner`, `ptgroup`, `ptaccess`, `ptsecurity`, `dataowner`, `datagroup`, `dataaccess`, `datasecurity`). Set this to an empty list to exclude nothing, or to a specific list of attribute names to exclude only those. | ### `excluded-attributes` Part of [`pi`](#pi) configuration. PI point attributes to exclude from being read into time series metadata. If left unset (default), a built-in list of internal PI attributes is excluded (`creationdate`, `creator`, `changedate`, `changer`, `ptowner`, `ptgroup`, `ptaccess`, `ptsecurity`, `dataowner`, `datagroup`, `dataaccess`, `datasecurity`). Set this to an empty list to exclude nothing, or to a specific list of attribute names to exclude only those. Each element of this list should be a string. ## `time-series` Global parameter. Include the `time-series` section for configuration related to the time series ingested by the extractor. This section is optional. The example would create time series on the following form in CDF: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi:12345", "name": "PI-Point-Name", "isString": false, "isStep": false, "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} external-id-prefix: 'pi:' external-id-source: SourceId data-set-id: 1234567890123456 ``` | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | Enter the external ID prefix to identify the time series in CDF. Leave empty for no prefix. The external ID in CDF will be this prefix followed by either the PI Point name or PI Point ID. | | `external-id-source` | either `Name` or `SourceId` | Enter the source of the external ID. `Name` means that the PI Point name is used, while `SourceID` means that the PI Point ID is used. Default value is `Name`. | | `data-set-id` | integer | Specify the data set to assign to all time series controlled by the extractor, both new and current. If you don't configure this, the extractor will not change the current time series' data set. | | `data-set-external-id` | string | Specify the external ID of the data set to use, see `data-set-id`. Using this requires the dataSets:READ ACL in CDF | | `sanitation` | either `Remove`, `Clean` or `None` | Specify what to do when the time series fields exceed CDF limits. `Remove` will skip any time series that fail sanitation. `Clean` will truncate and remove values to conform to limits. `None` does nothing (requests may fail as a result). External IDs are never truncated, any time series exceeding CDF limits will be skipped to avoid external ID collisions, regardless of this configuration. Default value is `Clean`. | | `update-metadata` | boolean | Enable updating time series if extractor configuration or PI metadata changes. Default value is `True`. | | `space-id` | string | The identifier for the Data modeling space. This extractor configuration parameter overrides `data-set-id` parameter, and changes the extractor destination from the Asset-Centric TimeSeries to a Core Data Model based TimeSeries, using the `CogniteExtractorTimeSeries` system model extension. | | `source-id` | string | Data modeling source id, this parameter will override the external id of the source node in data modeling, which defaults to the `host` parameter from PI server configuration. | ## `events` Global parameter. Configuration for writing events on reconnect and data loss incidents The example configuration produces events on the following form ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "externalId": "pi-events:PiExtractor-2020-08-04 01:01:21.395(0)", "startTime": 1596502850668, "endTime": 1596502880393, "type": "DataLoss", "subtype": "DataPipeOverflow", "source": "PiExtractor", "dataSetId": 1234567890123456 } ``` **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} source: PiExtractor external-id-prefix: 'pi-events:' data-set-id: 1234567890123456 store-extractor-events-interval: 5m ``` | Parameter | Type | Description | | - | - | - | | `source` | string | Events have this value as the `source` in CDF. | | `external-id-prefix` | string | Set an external ID prefix to identify events in CDF. Leave empty for no prefix. | | `data-set-id` | integer | Select a data set to assign to all events created by the extractor. We recommend using the same data set ID used in the [Time series](#time-series) section. | | `data-set-external-id` | string | Data set external ID added to events written to CDF, see `data-set-id`. Using this requires the `dataSets:READ` ACL in CDF | | `store-extractor-events-interval` | string | Store events in CDF with this interval. Format is as given in [Intervals](#intervals). If this is not set, events are not created in CDF.

    **Examples:**
    `10s`
    `5m` | ##
    `extractor` Global parameter. The `extractor` section contains various configuration options for the operation of the extractor itself. The options here can be used to extract only a subset of the PI points in the server. This is how the list is created: 1. If `include-tags`, `include-prefixes`, `include-patterns` or `include-attribute-values` are not empty, start with the union of these three. Otherwise, start with all points. 2. Remove points as specified by `exclude-tags`, `exclude-prefixes`, `exclude-patterns` and `exclude-attribute-values`. | Parameter | Type | Description | | - | - | - | | `attribute-filter-case-insensitive` | boolean | Enable case-insensitive comparison for attribute keys and values in include-attribute-values and exclude-attribute-values filters. When enabled, attribute value 'ABC' will match 'abc', 'Abc', 'ABC', etc. NOTE: This setting does NOT affect tag name matching (include-tags, exclude-tags), prefix matching (include-prefixes, exclude-prefixes), or pattern matching (include-patterns, exclude-patterns), which remain case-sensitive. Default is false (case-sensitive matching). | | [`include-tags`](#extractor.include-tags) | list | Include tags with name exactly equal to one of these names. | | [`include-prefixes`](#extractor.include-prefixes) | list | Include tags with name starting with one of these prefixes. | | [`include-patterns`](#extractor.include-patterns) | list | Include tags with name containing one of these substrings. | | [`include-attribute-values`](#extractor.include-attribute-values) | list | Include tags with attributes equal to at least one of these key/value pairs. | | [`exclude-tags`](#extractor.exclude-tags) | list | Exclude tags with name exactly equal to one of these names. | | [`exclude-prefixes`](#extractor.exclude-prefixes) | list | Exclude tags with name starting with one of these prefixes. | | [`exclude-patterns`](#extractor.exclude-patterns) | list | Exclude tags with name containing one of these substrings. | | [`exclude-attribute-values`](#extractor.exclude-attribute-values) | list | Exclude tags with attributes equal to at least one of these key/value pairs. | | `time-series-update-interval` | either string or integer | Interval between PI tag scans. Each scan discovers new PI tags and detects tags removed from PI but still present in CDF; removed tags are handled per extractor.deleted-time-series.behavior (applies to both classic and CDM time series, plus RAW metadata when enabled). The accepted formats for this field is either in the form of intervals like `4d`, or in the form of a cron expression like `0 4 * * *`. Default value is `24h`. | | [`deleted-time-series`](#extractor.deleted-time-series) | object | Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). | | `end-of-stream-interval` | string | Interval for fetching the end-of-stream timestamps from PI. Format is as given in [Intervals](#intervals). | | `end-of-stream-chunking` | integer | Maximum number of time series per end-of-stream request. Default value is `10000`. | | `end-of-stream-throttle` | integer | Maximum number of parallel end-of-stream requests. Default value is `10`. | | `time-series-update-throttle` | integer | Maximum number of parallel time series updates. Default value is `10`. | | `dry-run` | boolean | Set this to `true` to run the extractor in dry-run mode, where it does not push any data to CDF. Useful for testing the connection to the PI Server | | `read-extracted-ranges` | boolean | Set this to `false` to disable reading from CDF on extractor startup. If this is set to `false`, it is strongly recommended to have a state-store configured, or the extractor will read *all* history from PI on every run. Default value is `True`. | | [`status-codes`](#extractor.status-codes) | object | Configuration for ingesting status codes to CDF timeseries. | ### `include-tags` Part of [`extractor`](#extractor) configuration. Include tags with name exactly equal to one of these names. Each element of this list should be a string. ### `include-prefixes` Part of [`extractor`](#extractor) configuration. Include tags with name starting with one of these prefixes. Each element of this list should be a string. ### `include-patterns` Part of [`extractor`](#extractor) configuration. Include tags with name containing one of these substrings. Each element of this list should be a string. ### `include-attribute-values` Part of [`extractor`](#extractor) configuration. Include tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `exclude-tags` Part of [`extractor`](#extractor) configuration. Exclude tags with name exactly equal to one of these names. Each element of this list should be a string. ### `exclude-prefixes` Part of [`extractor`](#extractor) configuration. Exclude tags with name starting with one of these prefixes. Each element of this list should be a string. ### `exclude-patterns` Part of [`extractor`](#extractor) configuration. Exclude tags with name containing one of these substrings. Each element of this list should be a string. ### `exclude-attribute-values` Part of [`extractor`](#extractor) configuration. Exclude tags with attributes equal to at least one of these key/value pairs. | Parameter | Type | Description | | - | - | - | | `key` | string | Attribute name | | `value` | string | Attribute value | ### `deleted-time-series` Part of [`extractor`](#extractor) configuration. Include the `deleted-time-series` subsection to configure how the extractor handles time series that exist in CDF but not in PI. This subsection is optional, and the default behavior is `none` (do nothing). This only affects time series with the same data set ID and external ID prefix as the [time series configured in the extractor](#time-series) To find time series that exist in CDF but not in PI, the extractor: * Lists all time series in CDF that have the configured external ID prefix and data set ID * Filters the time series using the include/exclude rules defined in the [extractor](#extractor) section. * Matches the result against the time series obtained from the PI Server after filtering these using the include/exclude rules. | Parameter | Type | Description | | - | - | - | | `behavior` | either `None`, `Flag` or `Delete` | Select the action taken by the extractor. Setting this to `Flag` will perform soft deletion: Flag the time series as deleted but don't delete them from CDF. Setting it to `Delete` will delete the time series from CDF.

    **If you set this to `Delete`, the time series in CDF that cannot be found in PI will be permanently deleted from CDF.**. Default value is `None`. | | `flag-name` | string | If you've set `behavior` to `Flag`, use this string to mark the time series as deleted. Metadata is added to the time series with this as the key, and the current time as the value. Default value is `deletedByExtractor`. | | `time-series-delete-throttle` | integer | Maximum number of parallel delete operations. Default value is `10`. | ###
    `status-codes` Part of [`extractor`](#extractor) configuration. Configuration for ingesting status codes to CDF timeseries. | Parameter | Type | Description | | - | - | - | | `status-codes-to-ingest` | either `GoodOnly`, `Uncertain` or `All` | Which data points to ingest to CDF. `All` ingests all datapoints, including bad. `Uncertain` ingests good and uncertain data points. `GoodOnly` ingest only good datapoints. Default value is `GoodOnly`. | ## `backfill` Global parameter. Include the `backfill` section to configure how the extractor fills in historical data back in time with respect to the first data point in CDF. The backfill process completes when all the data points in the PI Data Archive are sent to CDF or when the extractor reaches the target timestamp for all time series if the `to` parameter is set. | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set to `true` to disable backfill. | | `step-size-hours` | integer | Step, in whole number of hours. Set to 0 to freely backfill all time series. Each iteration of backfill will backfill all time series to the next step before stepping further backward. This helps even out the progression of long backfill processes. Default value is `168`. | | `to` | string | The target CDF timestamp in miliseconds at which to stop the backfill. Format is as given in [Timestamps and intervals](#timestamps-and-intervals), `-ago` can be added to make a relative timestamp in the past. Default value is `0`.

    **Examples:**
    `3d-ago`
    `2025-07-02T22:23:12Z`
    `2022-11-20+12:00` | | `time-series-delay` | integer | Delay in milliseconds between each time series backfill request within a step. | | `step-delay` | integer | Delay in milliseconds between each step. | | `retry-bucket-size` | integer | Maximum size of the retry bucket. Zero or less for no limit. Default value is `100`. | | `retry-bucket-cleanup-time` | integer | Delay in seconds between each cleanup of the retry bucket. Default value is `3600`. | ##
    `frontfill` Global parameter. Include the `frontfill` section to configure how the extractor fills in historical data forward in time with respect to the last data point in CDF. At startup, the extractor fills in the gap between the last data point in CDF and the last data point in PI by querying the archived data in the PI Data Archive. After that, the extractor only receives data streamed through the PI Data Pipe. These are real-time changes made to the time series in PI before archiving. When data points are archived in PI, they may be subject to compression, reducing the total amount of data points in a time series. Therefore, the backfill and frontfill tasks will receive data points after compression, while the streaming task will receive data points before compression. Learn more about compression in [this video](https://www.youtube.com/watch?v=6-scv3oQ7Kk). | Parameter | Type | Description | | - | - | - | | `skip` | boolean | Set this to `true` to disable frontfill and streaming. | | `streaming-interval` | string | Interval between each call to the PI Data Pipe to fetch new data. Format is as given in [Intervals](#intervals). If set to zero, the extractor will not fetch live data from PI.

    **If you set this parameter to a high value, there is a higher chance of having a client outbound queue overflow in the PI server. Overflow may cause loss of data points in CDF.**

    Recommended range is between 1 and 30 seconds. If you observe data loss events, consider turning this value down. Default value is `1s`. | | `delete-data-points` | boolean | If you set this to `true`, the corresponding data points are deleted in CDF when data point deletion events are received in the PI Data Pipe.

    **Enabling this parameter may increase the streaming latency of the extractor since the extractor verifies the data point deletion before proceeding with the insertions.** | | `use-data-pipe` | boolean | Older PI servers may not support data pipes. If that's the case, set this value to `false` to disable data pipe streaming. The frontfiller task will run constantly and will frequently query the PI Data Archive for new data points. Default value is `True`. | | `time-series-chunk` | integer | The maximum number of time series in each frontfill query to PI. Value of this parameter cannot be larger than the value of `data-points-chunk`. Default value is `1000`. | | `data-points-chunk` | integer | The maximum number of requested data points in each frontfill query to PI. Default value is `10000`. | | `treat-missing-as-no-data` | boolean | If true, when PI returns no data points for a tag during frontfill, the extractor advances that tag's cursor to the current time instead of leaving it unmoved. Only applicable when use-data-pipe is false, will be a no-op when use-data-pipe is true.

    WARNING: enabling this will skip any historical data that arrives after the cursor has been advanced. Only enable this if you are certain the historical data won't arrive later. | ##
    `high-availability` Global parameter. Configuration for a Redis based high availability store. Requires Redis to be configured in [`state-store`](#state-store). | Parameter | Type | Description | | - | - | - | | `interval` | string | Interval to update the high availability state in Redis. Format is as given in [Intervals](#intervals). | | `timeout` | integer | Timeout in seconds before taking over as active extractor. Must be set greater than 0 to enable high availability. | # Extractor metrics Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/pi_metrics Monitor and debug the Cognite PI extractor using Prometheus metrics for performance, connection status, and data processing. The PI extractor can send performance metrics for remote monitoring and debugging. You can configure the PI extractor to use [Prometheus](https://prometheus.io). If you've enabled the [metrics section](/cdf/integration/guides/extraction/pi/pi#metrics), the extractor uploads metrics to the configured pushgateway or server. If you're using a pushgateway, all the metrics with descriptions will be listed under the configured job name. ## Prefix metric names The metric names are prefixed with the metric source: * `dotnet` is for metrics on .NET GC and memory. * `process` is for metrics about the running process, such as the number of threads, CPU time, and memory. * `push` is for metrics related to the pushgateway, such as last time pushed and last time failed. * `cognite_sdk` is for metrics produced by the Cognite .NET SDK. This is mostly related to the calls to the Cognite API. * `extractor_utils` is for metrics produced by the Extractor Utils library. This is related to the number and duration of calls to individual endpoints in CDF and metrics on state storage. * `pi_extractor` is for metrics specific to the operation of the PI extractor. ## Metrics | Name | Type | Purpose | | - | - | - | | pi\_extractor\_details | Gauge | Information about the version of the running extractor. The metrics contain the `version` label with the version of the running extractor. The actual metric value can be ignored.

    Example:
    `version="2.1.0.0"` | | pi\_extractor\_start | Gauge | The UNIX timestamp when the extractor was last started.

    Example:
    `1599495797230` (7 September 2020 16:23:17.230) | | pi\_extractor\_timeseries | Gauge | The total number of time series being processed by the extractor. | | pi\_extractor\_timeseries\_states | Gauge | The total number of time series per extraction state.
    The `state` label indicates whether the state is `frontfill`, `backfill`, or `streaming`. Example:
    `state="frontfill" value=13378` (13.000 time series are currently being frontfilled) | | pi\_extractor\_pi\_connection\_start | Gauge | The UNIX timestamp of the last connection attempt to the PI server.

    Example:
    `1599495797590` | | pi\_extractor\_pi\_connection\_time | Gauge | The elapsed time in milliseconds for the current connection to the PI server.

    Example:
    `7030683.7172` (\~1.95 hours) | | pi\_extractor\_pi\_connections | Gauge | The number of times the extractor has connected to the PI server.
    Internal errors on the extractor or on the PI server will cause the extractor to reconnect to the server, increasing the value of this gauge. If the PI extractor has been running since startup without reconnecting to the PI server, the value should be 1. | | pi\_extractor\_pi\_queries | Summary | The number and duration of queries to the PI server. This includes queries to the PI Data Archive (frontfill, backfill).

    **Sample count**: number of queries.
    **Sample sum**: Sum of the duration of all queries | | pi\_extractor\_pi\_query\_points | Counter | The total number of data points (AF values) returned in queries to the PI Data Archive (frontfill, backfill).

    There are two labels: `mode` and `state`.
  • `mode` is either `frontfill` or `backfill`.
  • `state` is either `good` or `bad`.


  • Example:
    `mode="frontfill" state="good" value=179070022` (\~180 million good data points received from `frontfill` queries) | | pi\_extractor\_pi\_data\_pipe\_events | Counter | The total number of events received from the PI Data Pipe (stream).

    There are three labels: `action`, `status` and `type`.

  • `action` is either `Add`, `Insert`, `Delete`, `Refresh`, or `Update`.
  • `status` is either `good` or `bad`.
  • `type` is either `AFDataPipeEvent` or `AFDataPipeRangeDeletedEvent` (C# class type of the event)


  • The total number of data points per (action, status, type).
    Example:
    `action="Update" status="good" type="AFDataPipeEvent" value=410045`
    (410.000 update events containing good data point updates) | | pi\_extractor\_pi\_data\_loss | Summary | The number and duration of data loss incidents observed in the PI Data Pipe (stream)

    **Sample count**: Number of data loss incidents observed.
    **Sample sum**: Sum of the duration of all incidents. | | pi\_extractor\_event\_queue\_size | Gauge | The number of events fetched from the PI Data Pipe and put in queues for processing.
    The `name` label indicates the type of the queue: `double` and `string`.

    The `double` queue contains data pipe events related to numeric data points.

    The `string` queue contains data pipe events related to string data points. Total number of events in the extractor queues.

    Example:
    `name="double" value=100000`
    (There are 100k numeric events in the queue to be processed by the extractor) | | pi\_extractor\_frontfill | Gauge | The maximum and minimum timestamps for the time series undergoing frontfill. The `aggregate` label indicates if it's the `max` or the `min` timestamp.
    Example:
    `aggregate="min" value=1546077427120` and `aggregate="max" value=1600256700000` (The minimum timestamp of all time series frontfilling is 29 December 2018 09:57:07.120 and the maximum is 16 September 2020 11:45:00. That is, there is about 2 year of data to be frontfilled). | | pi\_extractor\_backfill\_goal\_time | Gauge | The timestamp of the current backfill goal. The backfiller works in steps. This metric indicates how far back in time the backfiller will fetch data points for the current step. This can be used as a measure backfiller progress.

    Example:
    `value=1599652198896` (This timestamp is \~ UTC 9 September 2020. If the current UTC date is 16 September 2020, then the current backfill step is fetching data for one week back). | | pi\_extractor\_pusher\_data\_points | Counter | The total number of data points per action handled by the extractor **before** inserting into CDF.

    The `action` label specifies how the data point is being handled.

    `added` are points to be uploaded to CDF.

    `bad` are points discarded because they're marked as *bad* in PI.

    `frontfill-skipped` are data points skipped during frontfill.

    `backfill-skipped` are points skipped during backfill.

    Example:
    `action="added" value=100000`
    (there are 100k points to be added in CDF).To know the number of points that were actually uploaded to CDF, use extractor\_utils\_cdf\_datapoints | | pi\_extractor\_stream\_iterations | Counter | The number of stream iterations. During one iteration, the extractor streamer will dequeue events from the extractor event queues, parse the events, and upload the data points to CDF.

    Example:
    `value=150`
    (150 iterations since the extractor started). | | pi\_extractor\_stream\_iteration\_star | Gauge | The start timestamp of the current stream iteration. This can be used to calculate the streaming latency.

    Example:
    `value=1600265491419` (16 September 2020 14:11:31). Comparing it to UTC now (or even push\_time\_seconds) provides an indication of how late the current iteration is. | | pi\_extractor\_streamer\_data\_points | Counter | The number of data points per type being handled by the streamer.

    `label` type indicates the type of event in the PI server that generated the data point.

    `new` are real-time data being added to the time series (newer than the newest data point in a time series).

    `historical` are updates/insertion of data points in the past.

    `ahead` are new data points that are ahead of the newest data point in CDF because the frontfiller hasn't caught up with the real-time data points.

    Example:
    `type="new" value=100000`
    (100k new data points handled by the streamer so far.) | | pi\_extractor\_streamer\_oldest\_data\_point | Gauge | The age/time in seconds of the oldest (historical) data point received in the current stream iteration. This could be used as an indication of how far back the historical updates occurred, in case some updates are lost and a refill is required.
    Example:
    `value=120`
    (Oldest historical data point in the current streaming iteration is 2 minutes old.) | | extractor\_utils\_cdf\_timeseries\_requests | Summary | A summary describing the number and duration of time series requests to CDF. | | extractor\_utils\_cdf\_datapoint\_requests | Summary | A summary describing the number and duration of data point requests to CDF. | | extractor\_utils\_cdf\_datapoints | Counter | The total number of data points pushed to CDF by the utils. | | extractor\_utils\_cdf\_invalid\_data\_points | Counter | The total number of data points skipped due to bad timestamp or value. CDF requires data points to be within a certain time and value range. | | cognite\_sdk\_fetch\_inc | Counter | The total number of POST/GET/... actions performed | | cognite\_sdk\_fetch\_error\_inc | Counter | The total number of errors on actions. | | cognite\_sdk\_fetch\_retry\_inc | Counter | The total number of retries on actions. | | cognite\_sdk\_decode\_error\_inc | Counter | The total number of data errors, meaning invalid data was received from CDF. | | cognite\_sdk\_fetch\_latency\_update | Gauge | The measured latency in milliseconds on actions performed. | | dotnet\_total\_memory\_bytes | Gauge | The number of memory bytes of allocated memory | | dotnet\_collection\_count\_total | Counter | The number of garbage collection (GC) calls, grouped by generation. | | process\_cpu\_seconds\_total | Counter | The total number of CPU seconds used by the process. | | process\_num\_threads | Gauge | The number of current threads in use by the process. | | process\_open\_handles | Gauge | The number of currently open handles. | | process\_open\_fds | Gauge | The number of currently open file descriptors. | | process\_private\_memory\_bytes | Gauge | The number of current private memory bytes. | | process\_resident\_memory\_bytes | Gauge | The number of current resident memory bytes. | | process\_virtual\_memory\_bytes | Gauge | The number of current virtual memory. | | process\_working\_set\_bytes | Gauge | The number of memory bytes for the current working set. | | process\_start\_time\_seconds | Gauge | The start time in UNIX time seconds. | | push\_time\_seconds | Gauge | The time in UNIX time seconds data was last pushed to the Prometheus pushgateway. | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/pi_server_requirements Hardware, software, and network requirements for installing and running the Cognite PI extractor. You can install the extractor on a physical server, on a virtual machine on-premises, or in the cloud. ## Memory For the default values in the configuration, the server needs a minimum of 4 GB RAM and an additional 256 GB HD for logging and storage. You can adjust the configuration to reduce memory usage. ## CPU For the default values in the configuration, the server needs a minimum of **2-core CPUs**. The memory and CPU requirements also depend on the number of PI points in the PI Data Archive, the rate of real-time data points, the volume of historical data, and the rate at which the extractor checks for new PI points. ## Software The Cognite PI extractor is tested for AVEVA PI Server 2018 SP3. According to AVEVA's documentation, this implies that any SP3 Patch level will work. The extractor is also tested against Rockwell Automation's FactoryTalk Historian Site Edition (SE), which is built on the PI Server platform. * Windows Server 2016 or later. * A Windows account with admin rights to install the extractor. * Access to the Windows machine, for instance through VPN. * .Net Framework 4.8 Runtime or later. * PI AF SDK 2018 installed on the extractor host, distributed as part of the [PI AF Client 2018 SP3 Patch 3](https://docs.aveva.com/bundle/af-sdk/page/html/af-sdk-overview.htm). ## Network access * Username and password for connecting to the PI server where the data is stored. * Network access with firewall openings to the PI system. This means to each PI server on TCP port 5450. * Network access with firewall openings to the CDF cluster and CDF project. # Set up the PI extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/pi_setup Install and configure the Cognite PI extractor on Windows to stream time series data from OSISoft PI to Cognite Data Fusion (CDF). The Cognite PI extractor is distributed as a Windows installer (.msi file). When you run the .msi file, a setup wizard guides you through the installation process. The installer creates a Windows service and installs the binary files needed to run the extractor. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/pi/pi_server_requirements) for the extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#pi-extractor) in a CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). * Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime. * Download and run the installation files for the extractor from CDF. Set Modify permission under **Properties** in the installation folder to avoid permission issues. * Create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/pi/pi). The file must be in YAML format. ## Run as a Windows service Start the PI extractor service using the Windows service or with a command line: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} net start PiExtractor ``` The Windows service is named **Cognite Pi Extractor 4.3.0**, and the command line for the `net start` and `net stop` commands is named **PiExtractor**. Verify that the service is running. Running the Windows service Open the **Properties** for the service and set the **Startup type** to **Automatic (Delayed Start)**: Setting up the Windows service If you've configured the extractor to log messages to a file, you'll find the log files in the configured path, for instance, `logs/log.txt`. These should contain progress messages similar to the example below: Verify that the extractor is running If you've configured the extractor to push metrics, you can inspect the extractor operation and health in real-time using a Grafana board. ## Run the extractor behind a proxy Open and edit the `PiExtractor.exe.config` file from the installation directory. Add the `system.net` section under `configuration` with this content: ```xml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ``` Configure the proxy address and bypass rules according to the proxy server you want to use. Restart the extractor service. Read more about the [default-proxy-element](https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/defaultproxy-element-network-settings). ## Upgrade the extractor To upgrade the current extractor version, download and run the .msi file for the new version. The installer detects the current installation directory, updates the software, and keeps the configuration as-is. The sample configuration file is a part of the software package and is overwritten during an extractor upgrade. Download the PI extractor .msi file from CDF. Take note of the currently **installed version** in case of a rollback and of the **service properties** in the Properties dialog. During the upgrade, the installer resets all custom service properties to their default values, such as the service recovery settings. Run the installer. Update the custom service properties you noted in step 2. Verify that the extractor is running and that the version displayed in the `Starting PI Extractor` logline matches the upgraded version. Check the metrics and verify that the extractor resumes operation. If you've configured additional services to extract from multiple sources, the installer won't change the settings for these services during upgrades. If you've created multiple services, the following dialog appears during the upgrade: Upgrade extractor with multiple services The installer uses the default options, stops the services, and restarts the services after the upgrade. Otherwise, a server reboot is required to complete the upgrade process. ## Uninstall the extractor Run the installer and select **Remove the extractor** or open **Add or remove programs** from the control panel. Remove **Cognite PI extractor**. Optionally, remove the installation directory with the configuration file and logs. Optionally, delete the .msi file. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi/pi_troubleshooting Troubleshoot common issues with the Cognite PI extractor including authentication errors, connection problems, and data loss incidents. Inspect configuration errors with the Windows Event Viewer or check the configuration by manually starting the extractor from a Windows shell. ## The extractor is receiving 401 errors from CDF [HTTP error 401](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401) means that the service did not recognize the authentication the client provided. If the Cognite API returns 401 errors to the extractor, it means that the extractor was successful in getting an authentication token, but the token is not valid for the project in question. If this happens, it is usually because the configured `project` or `host` in the `cognite` section is incorrect. ## PI Server is not found Example log: ``` Connecting to PI Server (using AF SDK 2.10.9.593): ... pi server not found ``` If the extractor is failing to connect with an error message that the server is not found, try to make sure you can reach the server from the machine in question. Make sure the host is available, and that necessary ports are open. Tools like `ping` or `telnet` might be useful in debugging these connection issues. ## Extractor service errors You'll find error logs in the Windows Event Viewer under Windows Logs > Application. ### Config file not found When you install the extractor, the configuration file is not yet in the *config* folder. When the service starts, it fails with `Config file not found`. Prepare a [configuration file](/cdf/integration/guides/extraction/pi/pi), add it to the configuration folder and restart the service. ### Config version is not supported by this extractor If you use an older version of the configuration file, the error message `Config version is not supported by this extractor` appears. Replace the configuration file and restart the service. ### Failed to load config at line... Syntax errors in the configuration file will produce error events. See the [configuration settings](/cdf/integration/guides/extraction/pi/pi) for a list of all configuration options. ## Extractor upgrade errors ### Event 1000, Application error The extractor service stops after an upgrade with this error in the Windows logs. To resolve this, stop all extractor services, uninstall the extractor, install the new version, and restart the services. Extractor upgrade error message ## Data loss incidents Many data loss incidents are related to client-bound queue overflows: `[ERR] [PiSystem.PiConnection] Loss incident on the String data pipe: Overflow`. To resolve this, change the `MaxUpdateQueue` and `TotalUpdateQueue` PI server settings. If you still experience many data loss incidents (overflow or otherwise), see [PI Replace Utility](/cdf/integration/guides/extraction/pi_replace) for how to correct any data points inconsistencies in CDF. ## Requests to CDF time out Requests to CDF fail with this message: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [2021-06-25 06:02:21.188 WRN] [CogniteSdk.Client] Uploading data points to CDF timed out. Consider reducing the chunking sizes in the config file [2021-06-25 06:02:21.188 WRN] [CommonUtil.CDFDestination] Uploading data points to CDF timed out. Consider reducing the chunking sizes in the config file [2021-06-25 06:02:21.188 ERR] [PiExtractor.Extractor] Task failed: Streamer - One or more errors occurred. CommonUtil.DestinationException: Request to CDF timeout after retry ``` To resolve this, consider reducing the extractor chunking configuration. In particular, reduce the number of time series per data point insert requests: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite: cdf-chunking: data-point-time-series: 2000 ``` ## Cannot resolve remote name Requests to CDF fail with the following: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [DBG] [CogniteSdk.Client] Request timed out or failed with message: An error occurred while sending the request. Retrying in 125 ms. [DBG] [CogniteSdk.Client] Inner exception: The remote name could not be resolved: .... [ERR] [PiExtractor.Program] Unhandled exception: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote name could not be resolved: .... ``` To resolve, check if the extractor is running behind a proxy server and follow [these instructions](/cdf/integration/guides/extraction/pi/pi_setup#run-the-extractor-behind-a-proxy) to configure the extractor to use a proxy server. # About the Cognite PI AF extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_af Learn about the Cognite PI Asset Framework extractor and how it connects to OSIsoft PI AF to extract elements and their attributes and unit-of-measure classes into the CDF staging area (RAW). The extractor builds a hierarchy for each PI AF element in RAW, or you can configure the extractor to [create a flat structure in RAW](/cdf/integration/guides/extraction/configuration/piaf). You can filter the PI AF elements you want to extract using [PI queries](https://docs.osisoft.com/bundle/af-sdk/page/html/pipoint-query-syntax-overview.htm). # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_af/pi_af_server_requirements Hardware, software, and network requirements for installing and running the Cognite PI AF extractor. You can install the extractor on a physical server, on a virtual machine on-premises, or in the cloud. ## Memory For the default values in the configuration, the server needs a minimum of 4 GB RAM and an additional 256 GB HD for logging and storage. You can adjust the configuration to reduce memory usage. ## CPU For the default values in the configuration, the server needs a minimum of **2-core CPUs**. ## Software The Cognite PI AF extractor is tested for AVEVA PI AF Server 2018 SP3. According to AVEVA's documentation, this implies that any Patch level will work. The extractor is also tested against Rockwell Automation's FactoryTalk Historian Site Edition (SE), which is built on the PI Server platform. * Windows Server 2016 or later. * A Windows account with admin rights to install the extractor. * Access to the Windows machine, for instance through VPN. * .Net Framework 4.8 Runtime or later. * PI AF SDK 2018 installed on the extractor host, distributed as part of the [PI AF Client 2018 SP3 Patch 3](https://docs.aveva.com/bundle/af-sdk/page/html/af-sdk-overview.htm). ## Network access * Username and password for connecting to the PI AF server where the data is stored. * Network access with firewall openings to each PI AF server on TCP port 5457. * Network access with firewall openings to the CDF cluster and CDF project. # Set up the PI AF extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_af/pi_af_setup Install, configure, run, upgrade, and uninstall the Cognite PI AF extractor on Windows. The Cognite PI AF extractor is distributed as a Windows installer (.msi file). When you run the .msi file, a setup wizard guides you through the installation process. The installer creates a Windows service and installs the binary files needed to run the extractor. ## Before you start * Check the [server requirements](/cdf/integration/guides/extraction/pi_af/pi_af_server_requirements) for the PI AF extractor. * Assing [access capabilities](/cdf/access/guides/capabilities#pi-af-extractor) in a CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). * Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime. ## Install and configure the extractor Navigate to Data fusion > Integrate > Extractors and download and run the latest PI AF Extractor .msi file. Create a working directory. Set Modify permission under **Properties** in the installation folder to avoid permission issues. Create a `config` directory inside the working directory and create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/configuration/piaf). The file must be in YAML format. ## Run as a Windows service Start **PI AF Extractor service** from **Windows services**. Verify that the service is running. Open **Properties** for the service and set **Startup type** to **Automatic (Delayed Start)**. Verify that the extractor is running. If you've configured the extractor to [log messages to a file](/cdf/integration/guides/extraction/configuration/piaf#logger), check the log files in the configured path, for instance, *logs/log.txt*. If you haven't configured the extractor to read updates, it terminates after one reading. Check any error messages in the log to verify a successful run. ## Run from a command line You can run the extractor from a command line with these arguments: | Argument | Description | | - | - | | `--host` | Insert the PI server's host name or IP address. | | `--username` | Insert the PI server's Windows username. | | `--password` | Insert the PI server's Windows password. | | `--elements-query` | Specify the query parameter for PI AF elements. | | `-s`\|`--service` | Launch the extractor as a service. You must specify this parameter if you've configured a Windows service for the extractor. Running this from the command line only won't work. | | `-f` \|`--config-file` | Set the path to the configuration file. | | `-l`\|`--log-level` | Set the console log-level: `[fatal/error/warning/information/debug/verbose]` | | `-w`\|`--working-dir` | Set the working directory for the extractor. The default value is the current directory for a stand-alone file or one level above the executable file for the service version. | | `--log-dir` | Set the path to the log files to log to files unless you've already set the [Logger](/cdf/integration/guides/extraction/configuration/piaf#logger) configuration parameters. | | `-?`\|`-h+`\|--help | Enter to show this information. | ## Upgrade the extractor Download and run the .msi file for the new version of the extractor. The installer detects the current installation directory, updates the software, and keeps the configuration as-is. The sample configuration file is a part of the software package and is overwritten during an extractor upgrade. Download the PI AF extractor .msi file from CDF. Take note of the **installed version** in case of a rollback and of the **service properties** in the Properties dialog. During the upgrade, the installer resets all custom service properties to their default values, such as the service recovery settings. Run the installer. Update the custom service properties you noted in step 2. Verify that the extractor is running and that the version displayed in the *Starting PI Extractor* logline matches the upgraded version. ## Uninstall the extractor Run the installer, select **Remove the extractor** or open **Add or remove programs** from the Windows Control Panel. Remove the **Cognite PI AF extractor**. Optionally, remove the installation directory with the configuration file and logs. Optionally, delete the .msi file. # About the Cognite PI replace utility Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_replace Re-ingest time series data to CDF when data points are invalid, incorrect, or missing, and recover from data loss and reconnection incidents. The **Cognite PI extractor** can't synchronize PI content to CDF if the time series have invalid, incorrect, or missing data points. Use the **Cognite PI replace utility** to re-ingest time series to CDF by optionally deleting a data point time range and ingesting the data points in PI for that time range. By default, the PI replace utility re-ingests the entire time series, but you can limit this to a time range with the `start-time` and `end-time` parameters in the [configuration settings](/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration#replace). You can also configure the utility to [react to data loss and reconnection incidents](/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration) detected by the PI extractor. The PI replace utility doesn't modify the time series in CDF, only the data points. It assumes that all configured time series exist in PI and CDF and will skip any time series in PI that it doesn't find in CDF. Follow the PI replace progress through [logs and real-time metrics](/cdf/integration/guides/extraction/pi_replace/pi_replace_metrics) and see the execution operation summary report. ## Typical sources of inconsistencies in data points * **Downtime** on the PI extractor or loss of data between the PI AF SDK client and the PI server. The PI extractor can monitor these incidents. * The PI extractor **reconnects** to the PI server, for instance, when the extractor recovers from failures. During reconnection, the extractor may miss the notification from PI about historical updates or insertions of data points, which can cause missing or incorrect data points in CDF. * The client-bound queue in the PI server lost data. The typical cause is **queue overflow** due to bursts of data inserted and the client failing to fetch it. The dropped data never reaches the PI extractor, which can cause missing or incorrect data points in CDF. * Other sources of inconsistencies can be **bugs** in the PI extractor, causing incorrect data points in CDF. # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration Complete reference for configuring the PI replace utility including YAML settings for replace operations, data quality checks, events, and state storage. To configure the PI replace utility, you must create a configuration file. The file must be in [YAML](https://yaml.org) format. In addition to the sections listed below, you must include these PI extractor configuration sections: * [Cognite](/cdf/integration/guides/extraction/pi/pi#cognite) * [Time series](/cdf/integration/guides/extraction/pi/pi#time-series) * [PI](/cdf/integration/guides/extraction/pi/pi#pi) * [Logger](/cdf/integration/guides/extraction/pi/pi#logger) * [Metrics](/cdf/integration/guides/extraction/pi/pi#metrics) ## Minimal YAML configuration file The YAML settings below contain valid PI replace utility configurations, but not all configuration options are used. The values wrapped in `${}` are replaced with environment variables with that name. For example, `${COGNITE_PROJECT}` will be replaced with the value of the environment variable called `COGNITE_PROJECT`. The configuration file also contains the global parameter `version`, which holds the version of the configuration schema used in the configuration file. This document describes version 2 of the configuration schema. ```yml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} version: 2 cognite: project: ${COGNITE_PROJECT} idp-authentication: tenant: ${COGNITE_TENANT_ID} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_SCOPE} time-series: external-id-prefix: "pi:" pi: host: ${PI_HOST} username: ${PI_USER} password: ${PI_PASS} metrics: push-gateways: - host: "http://localhost:9091" job: "pi-replace-job" state-store: database: LiteDb location: "PiReplaceStateStore.db" logger: console: level: "information" replace: start-time: "2020-02-20 10:05:00" end-time: "2020-03-20 10:05:00" save-progress-interval: 10s status-codes: status-codes-to-ingest: All data-quality: check-data-quality: true report-file-name: "report" ignore-positive-difference: true ``` ## Configure the PI replace utility ### Replace | Parameter | Description | | - | - | | `start-time` | Run the PI replace utility starting from this time. You can specify the time using an ISO date string with one of these formats:
    `2020-02-20`, `2020-02-20 10:05:00` or `2020-02-20 10:05:00.321`.

    You can also specify the time as a time span using one of these formats:
    `360:10:00:00` (UTC now - 360 days and 10 hours), `360:10:00:00.000`, `10:00:00` (UTC now - 10 hours), or `10:00:00.000`.

    The default value is the minimum CDF timestamp: `1971-01-01 00:00:00` | | `end-time` | Run the PI replace utility ending at this time. You can specify the time using an ISO date string or a time span. See `start-time` for the allowed formats.

    The default value is UTC now. | | `startup-delay` | Insert the delay in seconds before the PI replace utility starts. This delay doesn't affect start and end times.

    The default value is 0. | | `delete-data-points` | If you set this to `true`, the data points in the range are deleted from CDF before inserting data points from PI.

    The default value is `false`. | | `throttling` | Enter the number of time series to replace concurrently. The default value is 10. | | `step-hours` | Replace time series in iterations going back in time in steps of this many hours. The default will replace in steps of 24 hours. Set this to zero or less to disable replacing in iterations (replace the whole time range in a single iteration). | | `save-progress-interval` | Enter an interval in seconds to save the PI replace progress to a state store. You must configure the selected storage in the [State store](#state-store) section.

    The default value is 10. | | `include-prefixes`, `include-tags`, `include-patterns`, `include-attribute-values` | Include time series based on these lists of tags (PI Point names), prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor.

    The default value is no value. | | `exclude-prefixes`, `exclude-tags`, `exclude-patterns`, `exclude-attribute-values` | Exclude time series based on these lists of tags, prefixes, patterns, and attribute values. These parameters are similar to the filters used by the PI extractor.

    The default value is no value. | | `dry-run` | Run the PI replace utility in `dry-run` mode. The replace utility compares the data points in CDF with the data points in PI, but doesn't upload any data points to CDF. The default value is `false`. | | `timeout` | Enter a timeout in seconds for each replace iteration for a time series, for instance, to make sure a single iteration doesn't take too long, forcing PI replace to stop.

    The default value is – 1 (disabled). | | [`status-codes`](#replace.status-codes) | Control which PI data points are ingested based on their status code. | #### Status codes By default, the PI replace utility ingests only good data points and silently skips bad and uncertain data points. Include the `status-codes` parameter in the [replace](#replace) section to control which data points from the PI source system are written to CDF. You can configure the PI replace utility to ingest good and uncertain data points, or all data points regardless of status, similar to the PI extractor's [`status-codes`](/cdf/integration/guides/extraction/pi/pi#extractor.status-codes) configuration. Requires PI extractor 4.21.0 or later. | Parameter | Description | | - | - | | `status-codes-to-ingest` | Controls which PI data points are ingested based on their status code. Set to `All` to ingest all data points regardless of status, `Uncertain` to ingest good and uncertain data points (bad data points are skipped), or `GoodOnly` to ingest only good data points. The default value is `GoodOnly`. | ### Data quality Include the `data_quality` section to compare the data points from PI with the data points in CDF to log and report any differences in the number of data points and the values of data points. If you've included the `metrics` section, the data quality metrics are sent to the configured [Prometheus pushgateway](https://prometheus.io). PI replace creates a CSV file with a summary by the end of the run. | Parameter | Description | | - | - | | `check-data-quality` | Set to `true` to enable data quality check. The replace utility compares data points from PI with data points in CDF and logs and reports differences. Data points are only replaced in CDF if differences are found. The default value is `false`. | | `report-file-name` | Insert the name of the CSV file used to write the quality check reports. Leave this parameter empty if you don't want to save the report to a file. The default value is `null`. | | `ignore-positive-difference` | If CDF has more data points than PI, and you've set this parameter to `true`, the data points from CDF aren't replaced with the data points from PI. The default value is `false`. | ### Events You can configure the Cognite PI extractor to [create CDF events for data loss and reconnection incidents](/cdf/integration/guides/extraction/pi/pi#events). These incidents may cause data points to be missing in CDF. Include the `events` section to fetch these events from CDF at startup and replace the data points for the time range when these events occurred. If you configure this section, the PI replace utility ignores the `start-time` and `end-time` parameters in the [replace](#replace) section. | Parameter | Description | | - | - | | `source` | Insert the name of the event source. This name must match the event source name configured in the PI extractor. The default value is `""`. | | `external-id-prefix` | Fetch only events with this external ID prefix. This setting should match the external ID configured in the PI extractor. The default value is `""`. | | `event-start-time` | Fetch events from CDF with a start time higher than or equal to this time. The allowed string formats are the same as the `start-time` parameter in the `replace` section. For example, 5:00:00:00 will fetch events starting 5 days ago. The default value is `""`. | | `event-end-time` | Fetch events from CDF with an end time less than or equal to this time. For example, 00:20:00 will fetch events ending 20 minutes ago. The default value is `""`. | | `expand-before` | Subtract this time span from the start time of the replace range (expand the range back in time). See the allowed time span formats in the `start-time` parameter in the replace section. The default value is `""`. | | `expand-after` | Add this time span to the end time of the replace range (expand the range forward in time). The default value is `""`. | ### State store Include the `state-store` and configure the `save_progress_interval` parameter in the [replace](#replace) section to store the PI replace progress. This is useful for long-running operations where a crash or restart may occur. PI replace will resume from where it stopped instead of starting again. | Parameter | Description | | - | - | | `database` | Enter the database you want to use. `LiteDb` creates a local file with the database. `Raw` will store data in CDF RAW. The default value is `none`. | | `location` | Insert the database file path for `LiteDb` or database name for `Raw`. The default value is `""`. | # Integrate with the Cognite PI extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_replace/pi_replace_integration Configure the PI extractor and PI replace utility to automatically detect and correct data loss and reconnection incidents using Cognite Data Fusion (CDF) events. The PI replace utility fetches the events at startup and replaces the data points for the time range when the events occurred. Set up the PI replace utility to [run daily](/cdf/integration/guides/extraction/run_extractor_win_task_scheduler) to find and correct data loss and reconnection incidents. ## Before you start * Install version 2.1.0 or higher of the Cognite PI extractor. ## Configure the PI extractor and PI replace utility to read CDF events To create CDF events each time the extractor detects a data loss incident or a reconnection, add the following section to the PI extractor configuration file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} events: source: 'PiExtractor' external-id-prefix: 'cog-pi-events.' data-set-id: 3258926993736049 store-extractor-events-interval: 10 ``` where: * `source` is a unique identifier for the PI extractor. * `external-id-prefix` is an optional prefix to the time series. * `data-set-id` is an optional data set for adding time series to. * `store-extractor-events-interval` determines how often or if the extractor pushes the events to CDF. The default value is -`1m`, which indicates that the extractor doesn't push events to CDF. Set a positive value to push to CDF. Here are two examples of events created by the PI extractor: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { 'externalId': 'cog-pi-events.Reconnection-2020-07-17 12:08:17.089(0)', 'dataSetId': 3258926993736049, 'startTime': 1594987632218, 'endTime': 1594987699519, 'type': 'Reconnection', 'subtype': 'ExtractorFailure', 'source': 'PiExtractor', 'id': 4894105677746187, 'lastUpdatedTime': 1595340918756, 'createdTime': 1594987732877, } ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { 'externalId': 'cog-pi-events.DataLoss-2020-07-17 12:14:55.523(1)', 'dataSetId': 3258926993736049, 'startTime': 1594987993792, 'endTime': 1594988095526, 'type': 'DataLoss', 'subtype': 'DataPipeOverflow', 'source': 'PiExtractor', 'id': 2759118070077116, 'lastUpdatedTime': 1595340918756, 'createdTime': 1594988119054, } ``` where: * `external ID` starts with the configured prefix, followed by the event type, the date and time the event occurred, and an event number. * `startTime` and `endTime` indicate when the incident started and ended. * `type` indicates if this is a reconnection or data loss incident. * `subtype` gives further information about the incident. For instance: * The `type` value `Reconnection` may indicate `ExtractorFailure` or `ExtractorRestart`. * The `type` value `DataLoss` may indicate `DataPipeOverflow` or `Other`. The PI replace configuration file must contain an [events](/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration#events) section with values identical to the [values in the PI extractor](/cdf/integration/guides/extraction/pi/pi#events): ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} events: source: 'PiExtractor' external-id-prefix: 'cog-pi-events.' event-start-time: '5:00:00:00' event-end-time: '00:20:00' expand-before: '05:00:00' expand-after: '00:10:00' ``` where: * `source` is required and must match the event source in the PI extractor. * `external-id-prefix` is optional. However, if it exists in the PI extractor, you should add this parameter. * `event-start-time` and `event-end-time` are the query range for events in CDF. In the example, PI replace will search for events that happened from 5 days ago to 20 minutes ago. * If PI replace finds events, the replace range is calculated based on the event's `startTime` and `endTime`, and expanded using the `expand-before` and `expand-after` configuration. In this example, the start time of PI replace will be 5 hours before the event `startTime`, and the end time will be 10 minutes after the event `endTime`. This figure illustrates how the replace range is calculated when two events are found. Replace range calculation diagram showing two events with expand-before and expand-after time spans, event start times, event end times, and the final replace range ## Updates and insertions of historical data points PI DataPipe values can be lost if the PI extractor is down. When the extractor restarts, it refills any gaps from stop to now (frontfill). However, the lost values can also be updates or insertions of historical data points, meaning they're already extracted. The extractor isn't able to recover these values. In the case of data loss incidents reported by the PI server, such as data pipe buffer overflows, the extractor can lose real-time and historical updates. The **Events Range** in the figure above covers the extractor downtime and the data loss period but doesn't account for lost historical updates and insertions. Use the `expand-before` parameter to specify how far back PI replace will check for missing or wrong data points. The parameter settings depend on the PI setup. Determine the value for this parameter using the [pi\_extractor\_streamer\_oldest\_data\_point](/cdf/integration/guides/extraction/pi/pi_metrics#metrics) metrics from the PI extractor. This metric shows how old the oldest data point received by the PI extractor streamer is. In the image below, the oldest data point is 1.8 days old. Therefore, we can assume that typically the oldest historical update is two days old and set the parameter `expand-before` to `2:00:00:00`, indicating going back 2 days before the start of the incident. Metric graph showing the pi_extractor_streamer_oldest_data_point with the oldest data point at approximately 1.8 days old There is no guarantee that the oldest value lost during the incident is within the replace range. Use the `expand-before` configuration to mitigate the effects of a data loss. Still, this value should be low enough to allow PI replace to complete in a reasonable time. For instance, we don't recommend setting `expand-before` to a year. ## Processed events PI replace adds metadata that indicates start time and completion time for the replaced data points to the extractor events: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { 'externalId': 'cog-pi-events.Reconnection-2020-07-17 12:08:17.089(0)', 'dataSetId': 3258926993736049, 'startTime': 1594987632218, 'endTime': 1594987699519, 'type': 'Reconnection', 'subtype': 'ExtractorFailure', 'metadata': { 'pi-replace-started': '2020-07-21 13:54:09.392', 'pi-replace-completed': '2020-07-21 14:15:18.595', }, 'source': 'PiExtractor', 'id': 4894105677746187, 'lastUpdatedTime': 1595340918756, 'createdTime': 1594987732877, } ``` If the `pi-replace-completed` metadata is added to an event, this event will no longer be processed by PI replace. # Extractor logs, reports and metrics Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_replace/pi_replace_metrics Monitor PI replace utility performance with logger output, CSV reports, Prometheus metrics, and data quality checks for Cognite Data Fusion (CDF) data points. You can configure the PI replace utility to check the data point quality in CDF against the data points in the PI Data Archive. If you include the `data-quality` section, the differences between PI and CDF will be logged to the [configured logger](#logger-output), sent to a Prometheus pushgateway using [metrics](#metrics), and written to a CSV file on exit using the [report-file-name](/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration#data-quality) configuration parameter. ## Logger output PI replace logs the differences between the PI Data Archive and CDF with warning levels: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} [2020-07-30 08:15:01.938 INF] Pushing metrics to http://localhost:9091 with job name replace-pi-job [2020-07-30 08:15:02.445 INF] Connecting to PI server [2020-07-30 08:15:19.484 INF] Found 16549 time series in PI. The replace range is ("2020-07-28T00:00:00.0000000Z", "2020-07-29T00:00:00.0000000Z") [2020-07-30 08:15:19.532 INF] 16549/16549 time series have not been processed. [2020-07-30 08:15:19.551 INF] Progress will be stored with an interval of 10 secs. [2020-07-30 08:15:20.061 WRN] TimeSeries1 (Numeric) - datapoint number difference: -2/9 [2020-07-30 08:15:20.074 WRN] TimeSeries2 (Numeric) - datapoint number difference: -11/31 [2020-07-30 08:15:22.960 WRN] TimeSeries3 - datapoint number difference: -6065/12042 [2020-07-30 08:18:02.830 INF] Completed replacing 1000/16549 time series in range (2020-07-28 00:00:00.000, 2020-07-29 00:00:00.000) (163.2766988 secs) [2020-07-30 08:24:53.899 WRN] TimeSeries4 (Numeric) - datapoints with different values 3/7 [2020-07-30 08:26:28.255 WRN] TimeSeries5 has no datapoints in CDF. Should have 6 [2020-07-30 08:27:05.487 WRN] TimeSeries6 has no datapoints in CDF. Should have 6 [2020-07-30 08:41:00.468 INF] Completed replacing 2000/16549 time series in range (2020-07-28 00:00:00.000, 2020-07-29 00:00:00.000) (177.6383223 secs) [2020-07-30 09:06:44.673 WRN] TimeSeries7 (Numeric) - datapoints with different values 3/8 ``` where: * `TimeSeries1 (Numeric)` is the data point number difference: `-2/9` indicates 9 data points in the PI Data Archive, but 2 are missing from CDF. * `TimeSeries4 (Numeric)` is the data points' value difference. `3/7` indicates that both CDF and PI have 7 data points, but 3 data points have different values in CDF compared to the data points in the PI Data Archive. * `TimeSeries5` has no data points in CDF. For instance: `Should have 6` indicate that the time series `TimeSeries5` exists in CDF but has no data points. The same time series has 6 data points in the PI Data Archive. ## CSV report The PI replace utility writes a report with the differences between the PI Data Archive and CDF to a CSV file on exit: CSV report showing time series differences between PI Data Archive and CDF including range, data points, count difference, value difference, max delta, and average delta where: * `TimeSeries: Name` is the name of the time series. This is the PI Point name. * `RangeStart` indicates that data points with timestamps higher than or equal to this date and time were replaced. * `RangeEnd` indicates that data points with timestamps smaller than or equal to this date and time were replaced. * `TotalDataPoints` is the total number of data points in the PI Data Archive. * `CountDifference` is the number of data points in CDF subtracted from the number of points in the PI Data Archive. * `ValueDifference` is the number of data points in CDF that have different values than the data points in the PI Data Archive. * `MaxDelta` is the maximum difference of the numeric data point values. The individual data point delta percentage is calculated as the fraction: (Value in PI - Value in CDF) / Value in PI * `AvgDelta` is the average data point value difference. ## Metrics PI replace can send metrics to a [Prometheus pushgateway](https://prometheus.io). The metrics can be displayed in a Grafana dashboard and monitored by CDF. Prometheus metrics dashboard showing heartbeat, estimated time left, number of time series, data points, time range left to cover, average duration, and data quality metrics | Name | Description | | - | - | | Heartbeat | The time since the last metrics push from the PI replace process. | | Estimated time left | The estimated time left until all time series are processed, based on the average iteration duration. | | # Time series | The total number of time series being processed. | | # Data points | The total number of data points processed until now. | | Time range left to cover | The time range left to query PI for data points. When it reaches 0, all time series have been replaced. | | Avg duration per 1000 | The average time it takes to replace 1000 time series in one iteration. The `step-hours` configuration parameter defines the time span of an iteration. | | Data points with different values | The percentage of the processed data points with different values in CDF compared to the PI Data Archive. | | Data point count difference | The percentage of the data points in the PI Data Archive that are missing from CDF. | | # Extractor incidents | The number and type (data loss or reconnection) of extractor incidents being handled. | ## Replace progress Replace progress dashboard showing overall progress towards goal timestamp, number of time series replaced per iteration, and duration per 1000 time series | Name | Description | | - | - | | Replace progress | The overall progress. `Replace to` is the goal timestamp (`start-time` configuration parameter) and `Replace progress` is the current iteration timestamp (step). The replace process is completed when the yellow line meets the green line. | | Iteration - # of time series replaced | The number of time series replaced in the current iteration (step-hours) and the replacing rate (time series/sec). | | Iteration - Duration per 1000 | The duration of replacing 1000 time series in one iteration (step). The dashed line is the estimated time left until all time series are replaced. | ## Performance Performance metrics dashboard displaying CPU usage, memory consumption, CDF request counts and latency, and uploaded data points with upload rate | Name | Description | | - | - | | Process | The CPU usage and memory consumption of the PI replace process on the host machine. | | CDF Requests | The total number and latency of requests made to CDF. | | CDF Uploaded data points | The total number of data points uploaded to CDF and upload rate (data point/sec). | ## Data quality Data quality metrics showing total number of data points with different values, data point count differences, and numeric data point delta percentages | Name | Description | | - | - | | Data points with different values | The total number of data points with different values and their rate (points/sec). | | Data point count difference | The total number of data points in the PI Data Archive that are missing from CDF and its rate (points/sec). | | Numeric data point delta | The average and maximum difference between numeric data points in the PI Data Archive and CDF, as a percentage. A `MaxDelta` of 100% means that one time series had at least one data point where ABS (Value in PI - Value in CDF) / Value in PI greater than Value in PI. | ## PI server PI server metrics showing number of PI connections, query rate, query latency, and total number of good and bad AF values obtained from PI | Name | Description | | - | - | | # PI connections | The number of times PI replace establishes a connection with the PI server. If this value is greater than 1, an error caused PI replace to reconnect to PI. | | PI query rate | The rate of queries (query/sec) to the PI server and the query latency. | | PI AF Values | The total number of good and bad AF Values obtained from PI and its rate (value/sec). Typically, the PI extractor converts 'good' AF Values to data points and uploads these to CDF. | # Set up the PI replace utility Source: https://docs.cognite.com/cdf/integration/guides/extraction/pi_replace/pi_replace_setup Learn how to install and configure the Cognite PI replace utility to correct data loss and reconnection incidents in Cognite Data Fusion (CDF). The Cognite PI replace utility is distributed with the Cognite PI extractor's Windows installer (.msi file). When you run the .msi file, a setup wizard guides you through the installation process. ## Before you start The PI replace utility has the same server requirements and access capabilities as the PI extractor: * Check the [server requirements](/cdf/integration/guides/extraction/pi/pi_server_requirements) for the PI extractor. * Assign [access capabilities](/cdf/access/guides/capabilities#pi-replace-utility) in a CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc). ## Run the PI replace utility Navigate to **Data fusion** > **Integrate** > **Extractors** and download and run the latest PI extractor .msi. PI Replace is a part of the `PiExtractor.exe` application, run with the `replace` subcommand. For example, if the installation directory is `C:\Cognite\`, you can run Pi Replace as `C:\Cognite\PiExtractor\PiExtractor.exe replace`. Create a working directory. This shouldn't be the same as the installation directory. Create a `config` directory inside the working directory and create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/pi_replace/pi_replace_configuration). The file must be in YAML format. Run `PiExtractor.exe replace` from the working directory. You can override the default configuration directory with the `-f ` command line option. ## Run periodically using Windows Task Scheduler The PI replace utility is often required to run periodically, for instance, daily. You can set up Windows Task Scheduler according to [this procedure](/cdf/integration/guides/extraction/run_extractor_win_task_scheduler) for scheduled runs. # REST extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/rest Learn how to use the Cognite REST extractor to read data from REST APIs on a fixed schedule with custom mappings and pagination. The REST extractor uses the Cognite mapping language to implement custom pagination, incremental load, and to handle responses. See [custom mappings](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings) for details on writing your mappings. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#cognite-rest-extractor) to *create* a hosted REST extractor and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models in the target CDF project. You can use OpenID Connect and your existing identity provider (IdP) framework to securely manage access to CDF data. For more information, see [setup and administration for extractors](/cdf/integration/guides/extraction/admin_oidc). ## Deploy the extractor Navigate to Data fusion > Integrate > Extraction pipelines > Hosted extractors. Select Set up hosted extractor, choose Cognite REST extractor, then select Set up extractor. ## When to use the hosted REST extractor REST is a generic system for creating APIs over HTTP. The hosted Cognite REST extractor ingests data from a REST API on a fixed schedule with custom mappings and pagination. Use it when your API fits the following scope: * The extractor makes requests to a single host. The exception is that the host specified under `authentication` may be different. * The extractor sends request payloads as a valid JSON blob. * The extractor stores the `last_run` field as the singular state value between executions. This value indicates the most recent execution of the REST extractor in your project. Use the [Cognite Extractor Utils](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest) library and develop a custom REST extractor for your source system when you need to make requests to multiple hosts, combine source data or information in complex ways, or read data from CDF. Use the hosted REST extractor when you want to list information from a REST API and ingest it into CDF resources without building a custom extractor. ## Configure the extractor The extractor configuration consists of several parts required to extract from a REST source. * **Host** is the hostname or IP address the REST extractor connects to. Enter only the hostname, not the full URL. For example, for `https://api.cognite.com/v1/project/test/assets`, the host is `api.cognite.com`. * **Scheme** is the type of connection to use, either `http` or `https`. Most public APIs use `https`. * **Port** is the port on the source server. Note that `http` and `https` typically use standard ports `80` and `443`. Leave the port blank unless your source system indicates a different port. * **CA certificate** trusts the API server when it uses a self-signed or privately signed TLS certificate. Upload the public certificate of that certificate authority. This is not the same as an **authentication certificate**, which presents a client certificate to the API. * **Authentication** controls how the extractor authenticates to the API. Unlike manually setting **headers**, using the authentication option ensures that passwords and secrets are not visible when the source is read, and that they are stored with additional encryption. Choose one of the following: * Basic authentication, header value, query parameter, or client credentials. * Use an authentication certificate — present a client certificate to the API (mutual TLS). Upload an authentication certificate and an authentication certificate key in PEM or DER format. If the key is encrypted, enter the password for the certificate key. * Use basic authentication and certificate — username and password plus a client certificate when the API requires both. After you create a source, open it from Hosted extractors, go to Authentication details, and select Edit to update credentials or certificate files. To configure sources with the API, including `authCertificate`, see [Sources](/api-reference/concepts/20230101/hosted-extractors-sources). * **interval** represents how often the job should run. You can pick any of the options from the fixed list of intervals. * **path** is the *path* to the API endpoint portion of the URL. It does not include the host information or the query parameters. For `https://api/cognite.com/v1/projects/test/assets?externalId=test` the **path** portion will be `/v1/projects/test/assets`. * **method** is the HTTP *method* for the request. The default is `get`. Sending a JSON body is only permitted if the method is `post`. * **body** is the JSON body to send with `post` requests. When this contains information, the `Content-Type` header will also be added to the request. * **query** is a list of key/value pairs for the initial *query* portion of the URL. For example, when using `/v1/projects/test/assets?externalId=test&name=something`, the query is `{ "externalId": "test", "name": "something" }`. * **headers** is the list of valid header key/value pairs representing the initial headers sent with the request. * **incremental load** is used to build the initial request based on *previous* runs of the extractor. [Incremental load](/cdf/integration/guides/extraction/rest#incremental-load) describes the configuration details. Mapping for incremental load is applied to the extractor startup. Incremental load avoids reading the same data every time the extractor runs. * **pagination** is used during a single extractor run to paginate through data received from the source. [Pagination](/cdf/integration/guides/extraction/rest#pagination) describes the configuration details. The pagination mapping is applied once for each request after the first and is used to determine whether to run more queries. ## Mapping context Like other hosted extractors, all mappings involved with hosted extractors are passed a `context` object. This uses the following format: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "last_run": 12345678, // Timestamps since 01/01/1970 when this extractor last ran. "response": { "headers": { // List of headers in the last response. "key": "value" }, "status": 201 // Last response message, will always be 2xx. }, "request": { "path": "/last/path", // Last request path. "host": "myhost.com", // Last request host. "headers": { // List of headers in last request. "key": "value" }, "query": { // List of query parameters in last request. "key": "value" }, "method": "get", // Last request method. "body": {} // Last request body } } ``` `response` will be empty for `incremental load` mappings, since there is no prior response when those are run. ## Incremental load Incremental load is applied to extractor *startup*. It may apply to query parameters, headers, or the body. The incremental load mapping is given only `context` as described [in the mapping context](#mapping-context), without anything in the `response` section. If the incremental load is set to use `body`, the result of the mapping will *set* the next message body. If you want to only modify the body given in the extractor configuration, you can use a mapping on the following form: ``` { ...context.request.body, ...{ "set-this-field": "to-value" } } ``` If set to use `queryParam` or `header`, the result will overwrite only the configured query parameter or header value. Example, for the CDF Assets API: `type: body` ``` { "filter": { ...context.request.body.filter, "lastUpdatedTime": { "min": try_int(context.last_run, 0) } } } ``` ## Pagination Pagination is applied after each request and used to determine whether the extractor should make more requests to the source. The pagination mapping is given the `context` as described [in the mapping context](#mapping-context), and the response body in `body`, is converted to JSON. If a pagination request returns the *same* value as the previous request or `null`, the extractor will stop paginating. This usually means you need to make sure to return `null` from the mapping if a cursor or similar is `null`. If pagination is set to use `body`, the result of the mapping will *set* the next message body. If you want to just modify the body given in extractor configuration or in `incremental load`, you can create a mapping using the following format: ``` { ...context.request.body, ...{ "set-this-field": "to-value" } } ``` If set to use `queryParam` or `header`, the result will overwrite only the configured query parameter or header value. Example, for the CDF Assets API: `type: body` ``` if(body.nextCursor, { ...context.request.body, "cursor": body.nextCursor }, null) ``` This will set `cursor` in the body if `nextCursor` is set in the response and return `null` to end pagination otherwise. # About the Cognite SAP extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/sap Learn how the Cognite SAP extractor connects to SAP systems and ingests data into Cognite Data Fusion (CDF). **Deploy with the Cognite Toolkit:** The [Foundation deployment pack](/cdf/deploy/cdf_toolkit/references/packages/foundation) includes a `cdf_sap_extractor` module with ready-made extraction pipeline configs for SAP assets, equipment, and functional locations — use it instead of configuring the extractor pipeline by hand. Run the extractor once for single extractions or set up regular runs to keep up with additions to the SAP entities. You can [load data incrementally](/cdf/integration/guides/extraction/sap/sap_setup#load-data-incrementally-odata-only) if you only want to process new or updated rows since the last extraction. The extractor is available as a [Windows executable file](/cdf/integration/guides/extraction/sap/sap_setup#run-as-a-windows-executable-file), [Windows service](/cdf/integration/guides/extraction/sap/sap_setup#run-as-a-windows-service), or a [Linux executable file](/cdf/integration/guides/extraction/sap/sap_setup#run-as-a-linux-executable). # Extractor metrics Source: https://docs.cognite.com/cdf/integration/guides/extraction/sap/sap_metrics Available Prometheus metrics for monitoring and debugging the SAP extractor. ## Prometheus metrics This section contains the full list of available metrics from the extractor. For information on the different types of metrics, see the [Prometheus documentation](https://prometheus.io). *Counter* starts at zero and increases during an extractor run. | Name | Type | Purpose | | - | - | - | | sap\_extractor\_start\_time | Gauge | The time stamp (seconds) of when the extractor last started. | | sap\_extractor\_finish\_time | Gauge | The time stamp (seconds) of when the extractor last finished. | | sap\_extractor\_memory\_bytes | Gauge | The memory usage of the extractor in bytes. | | sap\_extractor\_cpu\_percent | Total | The CPU usage of the extractor (0 — 100). Notice that this value is summed across all threads, so the value might exceed 100. | | sap\_extractor\_rows\_fetched | Counter | The number of rows fetched from the SAP OData entity. | | sap\_extractor\_queries\_failed | Counter | The number of failed OData queries. | | sap\_extractor\_queries\_success | Counter | The number of successful queries. | # Server requirements Source: https://docs.cognite.com/cdf/integration/guides/extraction/sap/sap_server_requirements Hardware and software requirements for installing the SAP extractor. You can install the extractor on a physical server, on a virtual machine on-premises, or in the cloud. ## Memory For the recommended default values in the configuration, the server needs a **minimum of 1 GB RAM** and an additional **50 GB HD for logging and storage**. You can adjust the configuration to reduce memory usage. ## CPU For the recommended default values in the configuration, the server needs a minimum of **2-core CPUs**. ## Software Use a **Windows server (2019 or later)**, **Windows 10**, **Windows 11**, or **Linux** operating system. ## Network access Make sure you have **network access** with firewall openings to the SAP endpoints, the CDF cluster, and the CDF project. # Set up the SAP extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/sap/sap_setup Learn how to set up and configure the SAP extractor to connect to SAP endpoints and extract data to CDF. ## Before you start * Assign [access capabilities](/cdf/access/guides/capabilities#sap-extractor) for the extractor to *write* data to the respective CDF destination resources. * Set up the SAP endpoints you want to extract data from. * Check the [server requirements](/cdf/integration/guides/extraction/sap/sap_server_requirements) for the extractor. * Create a configuration file according to the [configuration settings](/cdf/integration/guides/extraction/configuration/sap). The file must be in YAML format. ## Connect to SAP The extractor supports two different protocols for connecting to SAP: OData and SOAP. The extractor connects to OData V2 and OData V4 endpoints in the [SAP NetWeaver Gateway](https://help.sap.com/saphelp_em92/helpdata/en/ec/aeea50ca692309e10000000a445394/content.htm?no_cache=true). SAP OData is available in multiple SAP ERP versions, such as: | SAP version | Description | | - | - | | SAP ERP 6.0 | We recommend the [SAP Gateway Service builder](https://help.sap.com/docs/SAP_NETWEAVER_AS_ABAP_751_IP/68bf513362174d54b58cddec28794093/cddd22512c312314e10000000a44176d.html), which automates and generates OData entities from the SAP standard data schemas. There are multiple ways of mapping SAP entities to OData entities. To ensure that schema mapping is done correctly on the entity level, use the **Import DDIC Structure** function for structures or database table that you expose through the OData service. [See this guide](https://blogs.sap.com/2021/05/06/a-step-by-step-process-to-create-odata-services-in-sap-sap-hana-system) for creating an OData service in SAP ERP 6.0. | | SAP S/4HANA OnPremise | We recommend using the standard OData endpoints delivered in the SAP S/4HANA installation. You'll find the endpoints at the [SAP Business Accelerator Hub](https://api.sap.com). To extract data using SAP OData predefined schemas, activate the standard endpoints on the [/IWFND/MAINT\_SERVICES](https://help.sap.com/doc/saphelp_nw75/7.5.5/en-US/bb/2bfe50645c741ae10000000a423f68/content.htm?no_cache=true) transaction. For instance, the standard endpoint for [SAP plant maintenance (PM) work orders](https://api.sap.com/api/OP_API_MAINTENANCEORDER_0002/overview). | | SAP S/4HANA Cloud | The OData endpoints are available inside the cloud communication scenarios predefined by SAP. You'll find the standard endpoints at the [SAP Business Accelerator Hub](https://api.sap.com). For instance, the standard endpoint for [SAP plant maintenance (PM) work orders](https://api.sap.com/api/CE_API_MAINTENANCEORDER_0002/overview). | See [the SAP documentation](https://help.sap.com/doc/saphelp_ssb/1.0/en-US/71/d07b52a3566f54e10000000a44176d/content.htm?no_cache=true) for more information about SAP OData endpoints. We recommend using OData to connect to and extract data from SAP. The extractor connects to [SAP ABAP Web Services](https://help.sap.com/doc/saphelp_nw74/7.4.16/en-us/48/52347a08e672d0e10000000a42189c/content.htm?no_cache=true) exposed through the SOAP protocol. Every published SAP Web service will have a corresponding XML-based description, accessible in the Web Service Description Language (a WSDL document). The extractor connects to the Web service by reading the definitions in the WSDL URL and triggering the defined operations. For more information about SAP ABAP Web Services, see [SAP's official documentation](https://help.sap.com/docs/SAP_NETWEAVER_750/684cffda9cbc4187ad7dad790b03b983/cf9fb513ced74bbf82cac2231a358086.html). ## Secure SAP connections with self-signed certificates The extractor uses HTTPS to securely connect to SAP OData services. If your SAP system uses an internal or self-signed certificate, update the extractor’s certificate bundle to include it so connections between SAP, the VM environment, and the CDF service remain trusted. ### Export the existing certificate bundle 1. Locate the public CA bundle in the `certifi` library: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import certifi # Get the path to the CA bundle ca_bundle_path = certifi.where() print(f"CA bundle located at: {ca_bundle_path}") ``` 2. Copy the file from the printed path and save it on your desktop as `cacert.pem`. 3. Add the SAP self-signed certificate 1. Obtain the SAP self-signed certificate in PEM format. 2. Open the certificate in a text editor. 3. Open `cacert.pem` from step 1. 4. Prepend the SAP certificate contents to the top of the file. 5. Save the updated file. Now, `cacert.pem` contains both public and SAP self-signed certificates. ### Configure the extractor 1. Set the environment variable `REQUESTS_CA_BUNDLE` to the path of the updated `cacert.pem`: ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} $env:REQUESTS_CA_BUNDLE="C:\Users\\Desktop\cacert.pem" ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} export REQUESTS_CA_BUNDLE=/home//Desktop/cacert.pem ``` 2. Run the extractor in the same shell where the environment variable is defined. The certificate must be manually added to the bundle on the VM to ensure both SAP and CDF connections work correctly. ## Run as a Windows executable file Navigate to Data fusion > Integrate > Extractors and find the SAP extractor's package for **Windows executable**. Download and decompress the zip file. Open a command line window and run the executable file with the configuration file as an argument. In this example, the configuration file is named *config.yml* and saved in the same folder as the executable file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} > .\sap_extractor_standalone-win32.exe .\config.yml ``` You stop the extractor by pressing **Ctrl+C** on your keyboard. The log file is stored in the [configured path](/cdf/integration/guides/extraction/configuration/sap#logger.file). ## Run as a Windows service Navigate to Data fusion > Integrate > **Extractors** and find the SAP extractor's installation package for **Windows service**. Download and decompress the zip file to the same directory as a configuration file. You must name the configuration file `config.yaml`. As an administrator, open up a command line window in the folder you placed the executable file and the configuration file. Run the following command: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} > .\sap_extractor_service-win32.exe install ``` Open **Services** in Windows and find the **Cognite SAP extractor** service. Right-click the service and select **Properties**. Configure the service according to your requirements. ## Run as a Linux executable Navigate to Data fusion > Integrate > **Extractors** and find the SAP extractor's package for **Linux executable**. Download and decompress the zip file. Open a command line window and run the executable file with the configuration file as an argument. In this example, the configuration file is named *config.yml* and saved in the same folder as the executable file: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} > ./sap_extractor--linux path/to/the/folder/config.yaml ``` ## Pagination When you extract data from SAP OData, use different pagination methods: no pagination, client-side pagination, and server-side pagination. Use the [pagination-type](/cdf/integration/guides/extraction/configuration/sap#endpoints) configuration setting to specify the pagination type to use when running full-load queries. Pagination is available when you extract data from SAP OData endpoints. No pagination means that the extractor fetches all data available from the SAP OData endpoint without using chunking logic on the server or client. Therefore, the SAP extractor may time out and return an error while waiting for the response from SAP. OData client-side pagination uses query parameters from the client to define a record offset. This limits the volume of data retrieved from the server. A client-side pagination request is built with the following URI parameters: * **\$top**: specifies the number of records to return in a single batch. The default and maximum value is 1,000 records. * **\$skip**: specifies the number of records to bypass (skip) from the total data set before returning the desired subset. For example, a query with `$skip=2000&$top=500` returns the fifth page of data, assuming there's data available and the page size is 500 records. The SAP server controls server-side pagination. The server generates a cursor to control the next batch of requests. This cursor represents a pointer to the start of the next page in the full data set and is returned to the calling SAP extractor. The server uses a `$skiptoken` value to resume pagination from the position identified by the cursor. The pagination type you use in your extractor configuration depends on the SAP OData endpoint implementation your extractor connects to. ## Load data incrementally (OData only) If the OData entities have an **incremental field**, you can set up the extractor to only process new or updated entities since the last extraction. For example, you can use the S/4HANA standard OData entity `Maintenance Order` and enter the field `LastChangeDateTime` in the [`incremental-field`](/cdf/integration/guides/extraction/configuration/sap#endpoints) configuration parameter for incremental delta queries to the CDF staging area. The incremental field in SAP must be an `Edm.DateTimeOffset` field—for example, `LastChangeDateTime` on the `Maintenance Order` entity. Your SAP OData endpoint must return records ordered by the incremental field in **ascending** order. The extractor updates the state store with the timestamp from the last processed row. If records arrive in descending order (latest records first), the last row contains the oldest timestamp and the state store never advances. On each run, the extractor then re-fetches data from that old date and places extra load on the SAP server. The extractor depends on client-side pagination for incremental load queries. When you configure the SAP extractor for a custom SAP OData endpoint, make sure your SAP implementation supports client-side pagination and the `$orderby` operation. See [Client-side pagination implementation](https://help.sap.com/doc/saphelp_ssb/1.0/en-US/71/d07b52a3566f54e10000000a44176d/content.htm?no_cache=true) for more information on the implementation provided by SAP. ## Schedule automatic runs To schedule automatic runs on Windows, you can [run the extractors in Windows Task Scheduler](/cdf/integration/guides/extraction/run_extractor_win_task_scheduler). To schedule automatic runs on Mac OS and Linux, use [cron expressions](https://en.wikipedia.org/wiki/Cron). To enter a new cron job, run `crontab -e` to edit the cron table file with the default system text editor. Here's the format for a job in the cron table: ```bash showLineNumbers theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ``` ## Attachments The extractor can extract attachments stored in SAP document frameworks, such as Generic Object Services (GOS), and ingest these into CDF Files. The extractor connects to the SAP OData endpoint [API\_CV\_ATTACHMENT\_SRV](https://api.sap.com/api/OP_API_CV_ATTACHMENT_SRV_0001/overview) and fetches files that are linked to a standard SAP OData entity, such as maintenance orders. See the [attachments](/cdf/integration/guides/extraction/configuration/sap#endpoints.attachments) configuration section. You can extract attachments when you've connected to SAP S/4HANA servers. # Troubleshooting Source: https://docs.cognite.com/cdf/integration/guides/extraction/sap/sap_troubleshooting Find steps to troubleshoot common issues while running the SAP extractor. ## Common connectivity issues Before starting the extraction routines, the SAP extractor performs a connection check to all configured endpoints. ### ConnectTimeout ``` requests.exceptions.ConnectTimeout: HTTPConnectionPool(host=gateway_url, port=port): Max retries exceeded with url: /sap/opu/odata/sap/YOURSERVICE ``` This error occurs when the SAP extractor can't reach the configured SAP gateway, and the connection ends in a timeout. To solve this error, make sure the extractor can access the SAP OData gateway instance, for instance via the firewall, VPN, or SAP WebDispatcher. ### 401 Logon Failed ``` RuntimeError: Could not connect to SAP Gateway https://yourgatewayurl : Status Code 401 Logon failed ``` This error occurs when the extractor can't authenticate to SAP. Review the `client`, `username`, and `password` configuration parameters. ### 403 No service found ``` RuntimeError: Could not connect to SAP Gateway https://yourgatewayurl: Status Code 403 No service found for namespace '', name 'YOURSERVICE', version '0001' ``` This error occurs when the endpoint configured doesn't exist in SAP. Review the configuration in the `endpoints` section. ### 403 Access Denied ``` RuntimeError: Could not connect to SAP Gateway: 403 Access Denied ``` This error occurs when an SAP user doesn't have access in the SAP system to fetch data from the configured endpoint. The SAP server administrator must grant the access. ### SSL certificate verification failed ``` requests.exceptions.SSLError: HTTPSConnectionPool(host='yourgatewayurl', port=443): Max retries exceeded with url: /sap/opu/odata/sap/YOURSERVICE (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate'))) ``` This error occurs when the VM hosting the extractor can't validate the SAP system's server certificate during the HTTPS connection. This may be because the SAP endpoint uses a self-signed or internally issued certificate that isn't present in the VM's trusted certificate store. See [Secure SAP connections with self-signed certificates](/cdf/integration/guides/extraction/sap/sap_setup#secure-sap-connections-with-self-signed-certificates). ## Common data extraction issues ### AttributeError: Entityset XXX not defined ``` AttributeError: EntitySet not defined in , ... ``` This error occurs when the configuration parameter `sap-entity` for the related endpoint configuration is incorrect. You must use a valid entity name. ### Maximum string length Error (OData endpoint) ``` /IWND/COM_CONSUMER122 Maximum stringh length (2,145,386,496 bytes) exceeded ``` This is a limitation in SAP OData endpoint related to the size of the OData object in the response. This error is likely to happen when running the extractor in full load mode (`incremental-field` is not configured). ### Extractor might enter in a "loop" state when using `incremental-field` (Custom OData endpoints) The SAP extractor uses OData $skip and $top operations to handle pagination. When setting up the SAP extractor towards a custom SAP OData endpoint, make sure that your SAP implementation supports pagination. ### Extractor fails with error message "Incremental field is not valid" (OData endpoint) The SAP extractor validates whether the `incremental-field` set in the endpoint configuration is a valid `Edm.DateTimeOffset` field. This is required to perform incremental loads with pagination towards SAP. ### State store timestamp does not advance (OData endpoint) This issue occurs when the state store holds an old timestamp after each extractor run. The OData service returns records in descending order by the `incremental-field` instead of ascending, so the SAP extractor writes the oldest timestamp from the last processed row to the state store. Each subsequent run re-fetches data from that old date and places extra load on the SAP server. To solve this issue, ensure records are returned in ascending order by the `incremental-field`: * **Custom OData endpoints:** Configure the endpoint to return records in ascending order. * **Standard S/4HANA OData services:** Work with your SAP administrator to update the service implementation. See [Load data incrementally](/cdf/integration/guides/extraction/sap/sap_setup#load-data-incrementally-odata-only) for more information. # Configure SSL certificates and proxy settings for extractors Source: https://docs.cognite.com/cdf/integration/guides/extraction/ssl_certificates_and_proxies Step-by-step guide to configure self-signed certificates, mTLS, and corporate proxy TLS interception for .NET and Python extractors. Cognite extractors often need extra TLS trust and proxy settings when they run on-premises or in cloud virtual machines (VMs). You configure trust between the extractor runtime, source systems, your identity provider (IdP), and Cognite Data Fusion (CDF). Use this procedure for these enterprise network scenarios: * **Self-signed server certificates** on source automation or database systems. * **Corporate proxies and firewalls** performing TLS inspection by injecting certificates signed by an internal Certificate Authority (CA). * **Mutual TLS (mTLS)** requiring client authentication certificates. * **Forward proxies** routing outbound traffic from the extractor host. ## Prerequisites Verify all of the following before you start: * You know which extractor you are running and whether it uses the .NET runtime or Python (`httpx` or `requests`). See [Identify your extractor runtime](#identify-your-extractor-runtime). * You have administrator or root access on the extractor host so you can update the certificate store or environment variables. * You have the CA or self-signed server certificate in `.cer`, `.crt`, `.pem`, or `.pfx` format. For mTLS, you also have the client certificate and private key. * The extractor is installed, and you can restart its Windows service, Linux process, or container. ## Identify your extractor runtime Extractors use different cryptographic trust stores and TLS libraries depending on their underlying runtime: | Extractor runtime | Example extractors | Trust mechanism | Key configuration | | - | - | - | - | | **.NET runtime** | Cognite PI extractor, Cognite PI AF extractor, Cognite OPC UA extractor (Windows) | Operating system certificate store | Windows Certificate Manager (`certlm.msc`) or Linux CA store | | **Python (`httpx`)** | Modern Python extractors and SDK clients | Environment variable or custom bundle | `SSL_CERT_FILE` | | **Python (`requests`)** | Python extractors using `requests` / `certifi` | Environment variable or custom bundle | `REQUESTS_CA_BUNDLE` | ## Configure trust for self-signed certificates and private CAs When a source system presents a self-signed certificate, the extractor must trust that certificate to establish an encrypted connection. The same trust is required if a corporate proxy inspects outbound TLS traffic using an internal CA. .NET-based extractors inherit certificate trust directly from the host operating system. ### Windows host Export or obtain the root CA or self-signed server certificate in `.cer`, `.crt`, or `.pfx` format. 1. Open the Start menu, search for `Manage computer certificates` (`certlm.msc`), and open it. 2. Expand **Trusted Root Certification Authorities** > **Certificates**. 3. Right-click **Certificates**, select **All Tasks** > **Import**, and follow the wizard to import your certificate file into the **Local Machine** store. Restart the extractor Windows service or console application to pick up the updated system trust store. ### Linux host Copy your CA certificate (`.crt` format) to the system store: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Ubuntu / Debian sudo cp internal-ca.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates # RHEL / Rocky Linux / CentOS sudo cp internal-ca.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust ``` Restart the extractor process or container. Python extractors rely on bundled CA certificates. To add trust for internal CAs or self-signed certificates, create a combined certificate bundle and point the runtime environment variable to it. 1. Locate the default `certifi` CA bundle or export your standard public certificates: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} python -c "import certifi; print(certifi.where())" ``` 2. Make a copy of the default `cacert.pem` file in a persistent directory accessible by the extractor service user. 3. Append your PEM-encoded internal CA or self-signed certificate to the end of the copied file: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Linux / macOS cat internal-ca.pem >> /opt/cognite/certificates/cacert.pem ``` ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Windows PowerShell Get-Content .\internal-ca.pem | Add-Content C:\Cognite\certificates\cacert.pem ``` Configure the environment variable that matches your extractor's HTTP library: Set `SSL_CERT_FILE` to the path of the combined bundle: ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Windows (System environment variable or PowerShell) [Environment]::SetEnvironmentVariable("SSL_CERT_FILE", "C:\Cognite\certificates\cacert.pem", "Machine") ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Linux (/etc/environment or systemd service unit) export SSL_CERT_FILE=/opt/cognite/certificates/cacert.pem ``` Set `REQUESTS_CA_BUNDLE` to the path of the combined bundle: ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Windows (System environment variable or PowerShell) [Environment]::SetEnvironmentVariable("REQUESTS_CA_BUNDLE", "C:\Cognite\certificates\cacert.pem", "Machine") ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Linux (/etc/environment or systemd service unit) export REQUESTS_CA_BUNDLE=/opt/cognite/certificates/cacert.pem ``` Ensure the service account running the extractor has read permissions for the custom `.pem` bundle file. ## Verify certificate trust After you restart the extractor, confirm it connects without certificate errors: 1. Start or restart the extractor. 2. Check the extractor log. A successful TLS handshake does not include `CERTIFICATE_VERIFY_FAILED`, `certificate verify failed`, or `self signed certificate`. 3. If errors persist, use [Troubleshooting certificate and connection errors](#troubleshooting-certificate-and-connection-errors). Do not leave certificate validation disabled in production. Flags such as `source.ignore-certificate-issues: true`, `source.auto-accept: true`, or `verify: false` expose telemetry and credentials to interception. Use these settings only for short diagnostic tests while you set up certificate trust. ## mTLS certificate practices Some source systems (such as secure OPC UA endpoints, REST APIs, or MQTT brokers) require mutual TLS, where the extractor presents a client certificate to prove its identity. Extractor YAML for client certificates is product-specific — for the Cognite OPC UA extractor, see [`x509-certificate`](/cdf/integration/guides/extraction/configuration/opcua#source.x509-certificate) in the configuration reference. * **Secret managers**: Store private keys and client certificates in a vault such as Azure Key Vault or AWS Secrets Manager when you can. Many extractors support `!keyvault ` in YAML. See the configuration reference for your extractor. * **File-based certificates**: If you use local file paths, restrict filesystem permissions so only the extractor service account can read the files. * **Certificate renewal**: Rotate certificates before they expire so extraction does not stop. ## Configure forward proxy settings If your network routes outbound internet traffic through an HTTP or HTTPS forward proxy to reach CDF APIs and your identity provider (such as Microsoft Entra ID), configure standard proxy environment variables. ### Standard proxy environment variables Most extractors honor standard proxy environment variables: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Linux / macOS export HTTP_PROXY="http://proxy.example.com:8080" export HTTPS_PROXY="http://proxy.example.com:8080" export NO_PROXY="localhost,127.0.0.1,.local,.internal-domain.com" ``` ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Windows PowerShell $env:HTTP_PROXY="http://proxy.example.com:8080" $env:HTTPS_PROXY="http://proxy.example.com:8080" $env:NO_PROXY="localhost,127.0.0.1,.local,.internal-domain.com" ``` Ensure that internal source system hostnames and local IP addresses are included in `NO_PROXY` so the extractor communicates directly with local data sources without routing through the external proxy. *** ## Troubleshooting certificate and connection errors **Cause**: The extractor runtime cannot verify the source system or proxy certificate against its trust store. **Solution**: 1. Check whether the source system uses a self-signed certificate or whether an intermediate proxy is intercepting the connection. 2. Export the root CA or self-signed certificate. 3. Add the certificate to the OS trust store (for .NET extractors) or configure `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` (for Python extractors). **Cause**: Corporate firewalls often generate dynamic certificates signed by an internal enterprise CA. The host OS might trust this CA, but Python extractors or containerized workloads may not. **Solution**: Export your company's root and intermediate CA certificates from the corporate trust store, append them to the extractor's certificate bundle, and set `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE`. **Cause**: Flags such as `source.ignore-certificate-issues: true`, `source.auto-accept: true`, or `verify: false` disable certificate validation and leave telemetry and credentials open to interception. **Solution**: Do not use these flags in production. Use them only for short diagnostic tests, then restore validation. See the Warning in [Verify certificate trust](#verify-certificate-trust). *** ## Related documentation * [Allowlist and network requirements](/cdf/admin/allowlist) * [OPC UA extractor troubleshooting](/cdf/integration/guides/extraction/opc_ua/opc_ua_troubleshooting) * [SAP extractor setup](/cdf/integration/guides/extraction/sap/sap_setup) # About Cognite WITSML extractor Source: https://docs.cognite.com/cdf/integration/guides/extraction/witsml Overview of the Cognite WITSML extractor that ingests wellsite data using SOAP and ETP protocols into CDF. The Cognite WITSML extractor ingests data using the [Wellsite Information Transfer Standard Markup Language](https://www.energistics.org/witsml-data-standards) (WITSML), [WITSML specification (version 1.4.1.1)](https://energistics.org/witsml-developers-users), into CDF. SOAP is typically used for historical data, while ETP is typically for real-time integrations, such as WITSML log objects. The extractor ingests WITSML [non-growing objects](/cdf/integration/guides/extraction/witsml/witsml_guide), such as `wellbore`, into [CDF RAW](/cdf/integration/guides/extraction/raw_explorer) and growing objects, such as WITSML logs, into the CDF [time series](/api-reference/concepts/20230101/time-series) or [sequences](/api-reference/concepts/20230101/sequences) services. WITSML growing objects are characterized by the following: * A relatively large number of occurrences of recurring data is indexed to time or depth. * Additional occurrences of recurring data are inserted or updated over time. This is what causes the data object to grow. You can configure the extractor using [extraction rules](/cdf/integration/guides/extraction/witsml/witsml_configuration#extraction-rules) that define what, how, and when to ingest WITSML data into CDF. ## Run the extractor The extractor is distributed as a [Docker container](/cdf/integration/guides/extraction/witsml/witsml_setup), and you set [a simple or advanced mode](/cdf/integration/guides/extraction/witsml/witsml_setup#deploy-the-extractor) for low or high data volume integrations. # About extraction pipelines Source: https://docs.cognite.com/cdf/integration/guides/interfaces/about_integrations Monitor data flow in Cognite Data Fusion (CDF) with extraction pipelines to catch failures and ensure reliable data integration. For the **Integrations** tool (public preview)—create an integration, complete the connection, and monitor **tasks**—see [About Integrations](/cdf/integration/guides/interfaces/about_unified_integrations). **Extraction pipelines** remains available during the transition to Integrations. By activating automatic email notifications, you'll catch any extractor failures or data flow interruptions and can start troubleshooting using the extraction pipeline documentation. You can create and monitor extraction pipelines while you're already working with CDF data sets by opening Data fusion > Data catalog. You can also create and view pipelines across several data sets from Data fusion > Integrate > **Extraction pipelines**. On the **Extraction pipeline** page, you'll get an overview of successful and failed pipeline runs, see when an extractor last communicated with CDF, and the last data sync with CDF. You can contact the pipeline owner and other relevant stakeholders using the contact info added for a pipeline. The **extraction pipelines documentation** helps you keep track of the data set, source systems, target database tables, and comprehensive free-text descriptions will guide you through the troubleshooting. In the **Configuration settings for extraction** section, you can create and edit the extractor configuration to test and verify the extractor settings, preferably in testing and staging environments. When applying the extractor configuration to a production environment, we recommend setting up [remote configuration files](/cdf/integration/guides/interfaces/configure_integrations) stored in the cloud using versioned files and continuous integration systems, such as [GitHub Actions](https://github.com/cognitedata/upload-config-action), or directly with the [Cognite API](/api-reference/concepts/20230101/api-description). You can set up extraction pipelines for all types of extractors, such as [data flows from Azure Data Factory](/cdf/integration/guides/interfaces/postgres_gateway/setup_data_factory#step-6-report-run-status-for-data-extractions) and the [Cognite extractors](/cdf/integration/concepts/extraction). Extraction pipelines overview showing successful and failed runs # About Integrations Source: https://docs.cognite.com/cdf/integration/guides/interfaces/about_unified_integrations Learn about Integrations in Cognite Data Fusion (CDF) for setting up and monitoring on-premises extractors. Integrations is in [public preview](/cdf/product_feature_status#public-preview) and may change. If you do not see Integrations in your project, contact [Cognite Support](/cdf/support) or your Cognite representative. **Integrations** is a CDF tool for setting up and monitoring **integrations** from Data fusion > Integrate > Integrations. An **integration** is a registered data path into CDF, such as a Cognite DB, File, or OPC UA extractor on your infrastructure. It is not the same as [data integration](/cdf/integration/concepts/about_data_pipelines). Creating an integration creates the integration resource only. It does **not** create an extraction pipeline. The **Extraction pipelines** experience remains available during the transition to Integrations. Do not use the Integrations path and the Extraction pipelines path for the same external ID at the same time. ## What you can do * Select Set up to open Extractors and choose a supported on-premises extractor. * Download and install the extractor, then complete the **connection**. When the connection succeeds, the extractor has finished startup and check-in with CDF. * On the Configuration tab, paste your local configuration file content. Remote configuration is keyed by the integration external ID. * After the connection succeeds, monitor **tasks** in Task log and open a row for Task run details. * Set email notifications from the integration menu with Edit notifications. * On the Errors tab, review structured failures by severity and type, and open the related **task**. * Trigger remote actions on the extractor from CDF, for example start or stop a task. The extractor picks up actions on the next check-in. ## Further reading * [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration) * [Configuration settings](/cdf/integration/guides/interfaces/configuration_settings) * [Monitor integrations](/cdf/integration/guides/interfaces/monitor_unified_integrations) * [Assign capabilities](/cdf/access/guides/capabilities#integrations) * [About extraction pipelines](/cdf/integration/guides/interfaces/about_integrations) # Set up extraction pipelines Source: https://docs.cognite.com/cdf/integration/guides/interfaces/add_integrations Create and configure extraction pipelines to monitor data integration into Cognite Data Fusion (CDF). You need to add data set, name, and external ID on the **Create extraction pipeline** page. You can add or edit additional information later on the **Extraction pipelines overview** page. ## Before you start * A [data set](/cdf/data_governance/guides/datasets/create_data_sets) must exist for the data you want to add to an extraction pipeline. * Navigate to Access and set any of these capabilities for **users**, **extractors**, and **third-party actors**, such as GitHub Actions:
    User Action Capability Description
    End-user Create and edit extraction pipelines extractionpipelines:write Gives access to create and edit individual pipelines and edit notification settings. Ensure that the pipeline has read access to the data set being used by the extraction pipeline.
    View extraction pipelines extractionpipelines:read Gives access to list and view metadata of the pipeline.
    Create and edit extraction configurations extractionconfigs:write Gives access to create and edit an extractor configuration in an extraction pipeline.
    View extraction configurations extractionconfigs:read Gives access to view an extractor configuration in an extraction pipeline.
    View extraction logs extractionruns:read Gives access to view run history reported by the extraction pipeline runs.
    Extractor Read extraction configurations extractionconfigs:read Gives access to read an extractor configuration from an extraction pipeline.
    Post extraction logs extractionruns:write Gives access to post run history reported by the extraction pipeline runs.
    Third-party actors Create and edit extraction pipelines extractionpipelines:write Gives access to create and edit individual pipelines and edit notification settings. Ensure that the pipeline has read access to the data set being used by the extraction pipeline.
    Create and edit extraction configurations extractionconfigs:write Gives access to create and edit the extractor configuration from an extraction pipeline.
    ## Create extraction pipelines Navigate to Data fusion > Integrate > **Extraction pipelines**, or Data fusion > **Data catalog**. Then select a data set and open the **Lineage** tab to add a pipeline to the **selected data set**. Select **Create extraction pipeline**, where you will be requested to fill in the mandatory fields for creating a pipeline. Select **Create** to open the **Extraction pipeline overview**. On this page, you can add additional information to give contexts and insights about the pipeline. You'll see successful or failed runs when the connected extractor starts ingesting data into CDF. See the extractors' configuration articles for setup. ## Enable email notifications Data owners and other stakeholders can receive email notifications about the extraction pipeline runs. The notifications are triggered when an extraction pipeline reports a **failed run to CDF** or an extraction pipeline with continuous data flow **stops communicating** with CDF. The notification is sent when a predefined time condition is reached. Under **Contacts**, enter the **email address** for the data owner. Optionally, add other contacts for the extraction pipeline. Turn on the **Notification** toggle. Select **Confirm**. Email notification settings showing contact fields and notification toggle Email notifications are only sent when an extraction pipeline status changes state or CDF has not registered any communication with the pipeline after a predefined time condition. This is to prevent multiple emails for ongoing incidents. For new incidents, emails are only sent for the first reported failed run and when the incident is resolved. Multiple reported failures in succession are ignored. ## Edit the extractor configuration file When you set up the Cognite extractors, you must create a configuration file that fits your requirements. Refer to the [extractor documentation](/cdf/integration/concepts/extraction) for details. You can create or edit the configuration in the **Configuration file for extractor** section to test and verify the settings, preferably in testing and staging environments. When applying the configuration to a production environment, we recommend setting up [remote configuration files](/cdf/integration/guides/interfaces/configure_integrations) stored in the cloud using versioned files and continuous integration systems, such as [GitHub Actions](https://github.com/cognitedata/upload-config-action), or directly with the [Cognite API](/api-reference/concepts/20230101/api-description). Select **Create configuration** to create a file or copy and paste an existing file onto the canvas. Make your changes and select **Publish** to save. The extractor now reads the configuration from CDF. Test and verify the changed settings in the upcoming extractor run. Deploy the changed settings in a production environment, for instance, by committing the configuration file to GitHub for versioning and a [continuous integration pipeline](/cdf/integration/guides/interfaces/configure_integrations) using GitHub Actions. ## Best practice for documenting extraction pipelines It is good practice to enter comprehensive information about a pipeline to simplify troubleshooting and administration of pipelines. The minimum information you need to record is **Data set**, **Name**, and **External ID**. Monitor the pipeline status by setting up **email notifications** for failed and interrupted pipelines and add **contact details** for the pipeline owner and other stakeholders. You'll find the switch for activating email notifications when you add a contact. Select the **schedule** set up in the extractor to document how often the extractor is expected to update the data in CDF and check the **Last connected** to make sure CDF and the extractor are communicating. It's useful to record the **source system name** and the **RAW database tables** to keep track of where your data is extracted from and ingested into. Information specific to your organization can be added using the metadata fields with key/value pairs. You can **enter all contexts and other insights** about the pipeline to **speed up troubleshooting issues**. Enter free text using the **Documentation** field for this purpose. This is displayed as a ReadMe section on the **Extraction pipeline overview** page. Make sure to keep this content updated at all times. You can format the text using [Markdown](https://www.markdownguide.org/cheat-sheet). # Configuration settings Source: https://docs.cognite.com/cdf/integration/guides/interfaces/configuration_settings Complete reference for the connection and application configuration extractors use to report to Integrations in Cognite Data Fusion (CDF). Extractors that report to Integrations read two configurations. The **connection configuration** says how to reach CDF and which integration to report to. The **application configuration** contains extractor-specific settings, as well as common options such as state storage, metrics, and logging. Integrations is in [public preview](/cdf/product_feature_status#public-preview) and may change. This page documents the extractor-side configuration for Integrations. For the API that stores and versions the application configuration, see [Configuration](/api-reference/concepts/20230101-beta/integrations-config). ## Connection configuration file The connection configuration file must be YAML or JSON. Pass it to the extractor with the `-c`/`--connection-config` command-line option. All keys use kebab-case. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} project: my-project base-url: https://westeurope-1.cognitedata.com integration: external-id: my-db-extractor authentication: type: client-credentials client-id: ${COGNITE_CLIENT_ID} client-secret: ${COGNITE_CLIENT_SECRET} token-url: https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token scopes: https://westeurope-1.cognitedata.com/.default ``` ### Base connection object The following parameters sit at the root of the connection configuration. | Parameter | Type | Description | | - | - | - | | `project` | string | **Required.** The CDF project name. | | `base-url` | string | **Required.** The base URL of the CDF cluster, for example `https://westeurope-1.cognitedata.com`. | | `integration` | object | **Required.** Identifies which integration this extractor reports to. See [`integration`](#integration). | | `authentication` | object | **Required.** Credentials the extractor uses to authenticate with CDF. See [`authentication`](#authentication). | | `connection` | object | Network connection settings. See [`connection`](#connection). | ### `integration` This object identifies the integration the extractor reports to. | Parameter | Type | Description | | - | - | - | | `external-id` | string | **Required.** The external ID of the integration in CDF that this extractor reports to. | ### `authentication` The extractor supports two authentication types, selected with the `type` key. These parameters are common to both types. | Parameter | Type | Description | | - | - | - | | `type` | either `client-credentials` or `client-certificate` | **Required.** Which authentication method to use. | | `client-id` | string | **Required.** The client ID to authenticate with. | | `scopes` | string | **Required.** A space-separated list of OAuth scopes to request, for example `https://westeurope-1.cognitedata.com/.default`. | If `type` is `client-credentials`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `client-secret` | string | **Required.** The client secret to authenticate with. | | `token-url` | string | **Required.** The URL to fetch tokens from. | | `resource` | string | Optional `resource` parameter to pass with token requests. | | `audience` | string | Optional `audience` parameter to pass with token requests. | If `type` is `client-certificate`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Path to the client certificate file. | | `password` | string | Password for the certificate file, if it is encrypted. | | `authority-url` | string | **Required.** The authentication authority URL. | ### `connection` Use these optional settings to control retries and SSL verification for requests to CDF. | Parameter | Type | Description | | - | - | - | | `retries` | object | Retry behavior for requests to CDF. See [`retries`](#retries). | | `ssl-certificates` | object | SSL verification settings. See [`ssl-certificates`](#ssl-certificates). | #### `retries` These parameters control how the extractor retries failed requests. | Parameter | Type | Description | | - | - | - | | `max-retries` | integer | Maximum number of retries for a failed request. Default value is `10`. | | `max-backoff` | time interval | Maximum backoff time between retries. See [Time interval format](#time-interval-format). Default value is `30s`. | | `timeout` | time interval | Request timeout. See [Time interval format](#time-interval-format). Default value is `30s`. | #### `ssl-certificates` These parameters control certificate verification. | Parameter | Type | Description | | - | - | - | | `verify` | boolean | Whether to verify SSL certificates. Default value is `true`. | | `allow-list` | list | List of certificate paths to trust in addition to the system store. | ## Time interval format Settings that accept a time interval, such as `max-backoff` and `timeout`, use a string with a number followed by a unit: `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). Examples are `30s` and `5m`. A plain number without a unit is interpreted as seconds. ## Environment variables Instead of a connection configuration file, an extractor can read its connection settings from environment variables. | Environment variable | Corresponds to | | - | - | | `COGNITE_PROJECT` | `project` | | `COGNITE_BASE_URL` | `base-url` | | `COGNITE_INTEGRATION` | `integration.external-id` | | `COGNITE_CLIENT_ID` | `authentication.client-id` | | `COGNITE_TOKEN_SCOPES` | `authentication.scopes` | | `COGNITE_CLIENT_SECRET` | `authentication.client-secret` (client-credentials only) | | `COGNITE_TOKEN_URL` | `authentication.token-url` (client-credentials only) | | `COGNITE_CLIENT_CERTIFICATE_PATH` | `authentication.path` (client-certificate only) | | `COGNITE_AUTHORITY_URL` | `authentication.authority-url` (client-certificate only) | Set `COGNITE_CLIENT_SECRET` to authenticate with client credentials, or `COGNITE_CLIENT_CERTIFICATE_PATH` to authenticate with a client certificate. The extractor picks the authentication method based on which of these two variables is set. ## Application configuration The application configuration holds settings specific to what the extractor does, such as which queries or tags to run. Every extractor that reports to Integrations also supports the following common settings. | Parameter | Type | Description | | - | - | - | | `state-store` | object | Where the extractor persists its state between runs. See [`state-store`](#state-store). | | `metrics` | object | Where the extractor sends performance metrics. See [`metrics`](#metrics). | | `log-handlers` | list | Where the extractor writes logs. See [`log-handlers`](#log-handlers). Default is a single console handler at the `info` level. | By default, the extractor fetches the active revision of the application configuration from the integration's [configuration revisions](/api-reference/concepts/20230101-beta/integrations-config). Use `-f`/`--force-local-config` to load the application configuration from a local file instead. See [Command line options](#command-line-options). ### `state-store` This object controls where the extractor persists its state between runs. | Parameter | Type | Description | | - | - | - | | `type` | either `raw` or `local` | **Required.** Where to store extractor state. | If `type` is `raw`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `database` | string | **Required.** The CDF RAW database to store state in. | | `table` | string | **Required.** The RAW table to store state in. | | `upload-interval` | time interval | How often to save state to RAW. Default value is `30s`. | If `type` is `local`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Path to a local JSON file to store state in. | | `save-interval` | time interval | How often to save state to the file. Default value is `30s`. | ### `metrics` You can send metrics to a Prometheus push gateway, to CDF as time series, or to a local Prometheus endpoint. | Parameter | Type | Description | | - | - | - | | `push-gateways` | list | One or more Prometheus push gateways to send metrics to. See [`push-gateways`](#push-gateways). | | `cognite` | object | Push metrics as CDF time series. See [`cognite`](#cognite). | | `server` | object | Expose metrics on a local Prometheus endpoint. See [`server`](#server). | #### `push-gateways` Each list item is a Prometheus push gateway. | Parameter | Type | Description | | - | - | - | | `host` | string | **Required.** URL of the push gateway. | | `job-name` | string | **Required.** Job name to push metrics under. | | `username` | string | Username for the push gateway, if it requires authentication. | | `password` | string | Password for the push gateway, if it requires authentication. | | `push-interval` | time interval | How often to push metrics. Default value is `30s`. | | `clear-after` | time interval | If set, clear the gateway this long after the extractor stops. | #### `cognite` These parameters control metrics stored as CDF time series. | Parameter | Type | Description | | - | - | - | | `external-id-prefix` | string | **Required.** Prefix used on the external IDs of the time series the extractor creates. | | `asset-name` | string | Name of an asset to attach the metric time series to. | | `asset-external-id` | string | External ID of an asset to attach the metric time series to. | | `data-set` | object | Data set (by `id` or `external-id`) to put the metric time series in. | | `push-interval` | time interval | How often to push metrics. Default value is `30s`. | #### `server` These parameters expose a local Prometheus endpoint. | Parameter | Type | Description | | - | - | - | | `port` | integer | Port to expose the Prometheus endpoint on. Default value is `9000`. | | `host` | string | Host to bind the Prometheus endpoint to. Default value is `0.0.0.0`. | ### `log-handlers` `log-handlers` is a list of one or more log handlers. Select each handler with the `type` key. If `type` is `console`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `level` | one of `critical`, `error`, `warning`, `info`, `debug` | **Required.** Minimum level to log to standard output. | If `type` is `file`, these parameters apply. | Parameter | Type | Description | | - | - | - | | `path` | string | **Required.** Path to the log file. | | `level` | one of `critical`, `error`, `warning`, `info`, `debug` | **Required.** Minimum level to log to the file. | | `retention` | integer | Number of daily log files to keep. Default value is `7`. | ## Command line options These options apply to extractors that report to Integrations. | Option | Description | | - | - | | `-c`, `--connection-config` | **Required.** Path to the connection configuration file. | | `-f`, `--force-local-config` | Path to a local application configuration file, used instead of fetching the configuration revision from CDF. | | `-l`, `--log-level` | Overrides the console log level from `log-handlers` for this run. One of `debug`, `info`, `warning`, `error`, `critical`. | | `--skip-init-checks` | Skips startup checks. Useful for debugging. Do not use in production. | | `--cwd` | Sets the working directory for the extractor. | | `--service` | Runs the extractor as a Windows service. Only supported on Windows. | | `--bootstrap-log-file` | Path to a log file that captures startup logs before the application configuration loads. Default value is `logs/bootstrap.log`. | ## Further reading * [Set up on-premises integrations](/cdf/integration/guides/interfaces/set_up_on_premises_integration) * [Monitor integrations](/cdf/integration/guides/interfaces/monitor_unified_integrations) * [Configuration](/api-reference/concepts/20230101-beta/integrations-config) # Set up extraction pipelines with remote configuration files Source: https://docs.cognite.com/cdf/integration/guides/interfaces/configure_integrations Configure extractors remotely using cloud-stored configuration files with GitHub Actions or the Cognite API. To deploy your extractors, you must supply a minimal configuration file containing the sign-in credentials and pull the remaining configuration from the cloud. You can generate the configuration files in the cloud with continuous integration systems such as [GitHub Actions](https://github.com/cognitedata/upload-config-action) or directly with the [Cognite API](/api-reference/concepts/20230101/api-description). If you created an **integration** in [Integrations](/cdf/integration/guides/interfaces/set_up_on_premises_integration) (public preview), paste your local configuration file on the integration Configuration tab, then validate and publish. Configuration is keyed by the integration external ID. You do not need the Extraction pipelines path below for that flow. The Extraction pipelines path remains valid if you still use extraction pipelines. ## Before you start Make sure the extractor has the `extractionconfigs:WRITE` capability to access CDF. ## Configure extractors with GitHub Actions Configure your extractor with a minimal configuration file. Refer to the [extractor documentation](/cdf/integration/concepts/extraction) for details. All the Cognite extractors have a similar configuration. This is an example for the Cognite DB extractor: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: remote cognite: host: ${BASE_URL} project: ${PROJECT} idp-authentication: client-id: ${CLIENT_ID} secret: ${CLIENT_SECRET} token-url: ${TOKEN_URL} scopes: - ${BASE_URL}/.default extraction-pipeline: external-id: db-extractor-pipeline ``` At startup, the extractor attempts to read the configuration files from the extraction pipeline with the external ID `pipeline-external-id`. The extractor continues to check for updates every few minutes. Create configuration files directly with the [Cognite API](/api-reference/concepts/20230101/api-description) or set up a continuous integration pipeline in [GitHub Actions](https://github.com/cognitedata/upload-config-action). To use the action, you must create a GitHub Workflow: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: Update extractor configuration on: push jobs: deploy-configs: runs-on: ubuntu-latest name: Deploy Configs steps: - uses: actions/checkout@v2 - name: get commit message id: commitmsg run: 'echo ::set-output name=commitmessage::$(git log --format=%B -n 1 ${{ github.event.after }})' - name: Deploy uses: cognitedata/upload-config-action@v1 with: base-url: ${{ secrets.BASE_URL }} token-url: ${{ secrets.TOKEN_URL }} cdf-project-name: ${{ secrets.PROJECT }} client-id: ${{ secrets.CLIENT_ID }} client-secret: ${{ secrets.CLIENT_SECRET }} root-folder: 'root_dir/' deploy: 'true' revision-message: '${{ steps.commitmsg.outputs.commitmessage }}' ``` Place the configuration files in the folder `root-folder`. The configuration file name must be identical to the external ID of the extraction pipeline. For instance, `db-extractor-pipeline.yml`. The extractor finds and runs the configuration file at startup. # Monitor extraction pipelines Source: https://docs.cognite.com/cdf/integration/guides/interfaces/monitor_integrations Monitor extraction pipelines to ensure reliable data integration and quickly identify and troubleshoot failures. Improve data quality management and make sure the data integration into Cognite Data Fusion (CDF) [data sets](/cdf/data_governance/concepts/datasets) is reliable by monitoring extraction pipelines. If any data integration issues occur, you can be quickly notified, start troubleshooting, and contact the pipeline owner and other stakeholders. ## View extraction pipeline details Navigate to Data fusion. * To view all extraction pipelines across data sets in a CDF project and their latest status, navigate to Integrate > Extraction pipelines. * To view extraction pipelines for the **selected data set**, navigate to Data catalog. Then select a data set, and open the **Lineage** tab. Select the pipeline name in the list to open the **Extraction pipeline overview**. Depending on the extractor, open the **Run history** tab (typical for on-premises extractors) or the **Insight** tab (typical for hosted extractors) to view run-level details. After you open a pipeline's details, use the following to interpret pipeline status, uptime, per-run messages, and where to investigate next. ## Understanding status and uptime The **Extraction pipeline** statuses reflect the latest known state of the pipeline and its runs. Treat the status as a signal to investigate. One failing step, such as authentication, connectivity, scheduling, or a failed upload to CDF, can show **Failure** or otherwise look unhealthy even when the rest of your integration is still healthy. Choose the tab that matches your extractor. On **Overview**, **Last reported status** summarizes successful and failed critical operations. **Last connected** shows when an extractor last communicated with CDF, and **Schedule** shows when the extractor is expected to run. On **Run history**, the **Message** column shows the errors or success summaries. Compare timestamps and **Configuration revision** across runs. Treat any failing status or stale timestamp as a signal to investigate. On **Overview**, each row in the jobs table is a **job** with an MQTT **topic filter** (the broker subscription in the **Topic filter** column). Each job has its own **Status** (for example **Running** or **Paused**). **Topic filters status** aggregates how reliably **all topic filters on this extractor** receive traffic over a recent window (uptime percentage and chart; broker subscription and ingestion health, not full validation in CDF). **Insight** or **View full insight** shows a deeper view of uptime and run-level signals. ## Troubleshooting failed integrations If an extraction pipeline fails or a continuous data flow into CDF is interrupted, you'll get an email from the system (if activated in setup). Select the email link to open the **Extraction pipeline overview**. Choose the tab that matches your extractor. * Select the pipeline name to open the **Extraction pipeline overview** tab to find documentation and key properties. * Use **Owner** and **Contacts** to see who to inform about a failure. * Review and confirm what is happening with **Last reported status**, **Last run time**, and **Last connected**. * Switch to the **Run history** tab to review past runs, filter by time and **Last run status**. * Review the chart, extractor metrics, and logs to get broader knowledge to see how the extraction process has performed over time. * When you have fixed the failure, update the pipeline documentation with any lessons learned. * Review extractor details, connection details, and recent errors. * Compare each job's **Status** with **Topic filters status** and review the uptime to measure availability in operations. * Use **Insight** (or **View full insight**) for charts and detail, and review extractor metrics or logs if data in CDF still looks wrong. # Monitor integrations Source: https://docs.cognite.com/cdf/integration/guides/interfaces/monitor_unified_integrations Monitor on-premises integrations in Cognite Data Fusion (CDF) using tasks, Task log, Task run details, and notifications. After an on-premises **integration** completes its **connection** (extractor startup and check-in), review **tasks** in Task log and open a row for Task run details. Integrations is in [public preview](/cdf/product_feature_status#public-preview) and may change. **Tasks** appear only after the connection succeeds. For Extraction pipelines, see [Monitor extraction pipelines](/cdf/integration/guides/interfaces/monitor_integrations). ## Open tasks for an integration Navigate to Data fusion > Integrate > Integrations. Select the on-premises integration. Use the Overview, Errors, and Configuration tabs as needed. On Overview, select a **task** by name. A task is a named unit of work from the extractor, such as a query or a continuous process. ## Review Task log On the task page, use Task log to list executions. Columns include Started, Finished, Status, Configuration, and Details. If Task log is empty, confirm the extractor is running and that the connection succeeded. ## Open Task run details Select a row in Task log, or select See more when it appears. In Task run details, review Details and Configuration. ## Task types | Task type | Meaning | | - | - | | Batch | Runs to completion in one execution, then finishes. Examples: a one-time startup task, or a scheduled run that exits each time. | | Continuous | Starts once when the extractor starts and keeps running for the extractor lifetime. Example: a persistent subscription loop. | ## Review errors On the integration Errors tab, extractors can report structured errors that you can search and filter. Each error can include: | Field | What it tells you | | - | - | | Severity | How serious the issue is: Warning, Error, or Fatal. | | Error type | The category of failure, such as general or configuration. | | Link to task | Which **task** reported the issue. | | Started / Finished | When the error began and ended, so you can see if it is still active. | Use severity and time range to find active issues, then open the linked **task** to inspect Task log and Task run details. ## Run remote actions On-premises extractors can declare actions that you trigger from CDF, such as start task, end task, or a custom action. CDF does not need inbound ports on customer infrastructure. The extractor picks up the action on the next check-in, with an optional message and metadata. Action lifecycle: | Status | Meaning | | - | - | | Pending | CDF queued the action. The extractor has not started it yet. | | Running | The extractor is running the action. | | Succeeded | The action completed successfully. | | Failed | The action did not complete successfully. | | Canceled | The action was canceled before it finished. | Navigate to Data fusion > Integrate > Integrations, then open the on-premises integration. Select the action the extractor exposes, for example start or end a **task**. Add a message or metadata if the UI offers those fields. Wait for the next extractor check-in, then confirm the action moves from Pending to Running, and then to Succeeded, Failed, or Canceled. ## Configuration On the Configuration tab, review or update remote configuration keyed by the integration external ID. ## Set up notifications Receive email notifications when a **task** fails or resumes regular activity, and when there is no activity for a set time. Open the integration. From the integration menu next to the name, select Edit notifications. On Subscriptions, select Add users to receive notifications and choose the users who should get email alerts. Select Inactivity threshold. Enter how long CDF should wait with no activity before it notifies the error log and subscribers. The value is in minutes (for example `1440`). ## Further reading * [About Integrations](/cdf/integration/guides/interfaces/about_unified_integrations) * [Set up an on-premises integration](/cdf/integration/guides/interfaces/set_up_on_premises_integration) * [Monitor extraction pipelines](/cdf/integration/guides/interfaces/monitor_integrations) # Setup and administration for PostgreSQL gateway 2.0 Source: https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway Set up the Cognite PostgreSQL gateway to ingest data into CDF from ETL tools using Microsoft Entra ID authentication. When you've registered the PostgreSQL gateway, users can sign in with their organizational ID to integrate data in a CDF project. This section describes how to set up a PostgreSQL gateway 2.0 instance. See the [migration guide](#migrate-to-postgresql-gateway-20) if you're migrating from a previous version. ## Before you start Make sure you have [registered the Cognite API and the CDF application in Microsoft Entra ID](/cdf/access/entra/guides/configure_cdf_azure_oidc) and [set up Microsoft Entra ID and CDF groups](/cdf/access/entra/guides/create_groups_oidc) to control access to CDF data. ## Step 1: Register an app in Microsoft Entra ID to use with PostgreSQL gateway Sign in to the [Azure portal](https://portal.azure.com) as an admin. If you have access to multiple tenants, use the Directory + subscription filter in the top menu to select the tenant in which you want to register an application. Search for and select **Microsoft Entra ID**. Under **Manage**, select **App registrations** > **New registrations**. In the **Register an application** window, enter the app name, and then select **Register**. Copy and make a note of the **Application (client) ID**. This value is required to create user credentials for the PostgreSQL gateway. 1. Under **Manage**, select **Certificates & secrets**. 2. Select **New client secret**. Certificates and secrets page showing new client secret option 3. Enter a client secret description. 4. Select an expiry time. 5. Select **Add**. Copy and make a note of the client secret in the **Value** field. Make sure you copy this value now. This value will be hidden after you leave this page. ## Step 2: Create a group in Microsoft Entra ID and add the registered app as its member Open the overview window in Microsoft Entra ID and select **Manage** > **Groups**. Create a group, read more [here](/cdf/access/entra/guides/create_groups_oidc). 1. Open the group. Under **Manage**, select **Members** > **Add members**. 2. Find the app you created above and click **Select**. Return to the overview, and then copy and make a note of the **Object Id**. Microsoft Entra ID group overview showing Object ID ## Step 3: Create a group in CDF and link to the Microsoft Entra ID group Sign in to CDF as an admin and naviate to Admin > Groups > Create new group. Create new group dialog in CDF In the **Create a new group** window, enter a **unique name** for the group. Add relevant capabilities. The **minimum** required capabilities are: * `postgresGateway:READ` * `postgresGateway:WRITE` * `project:list` * `groups:list` * `session:CREATE` The created user will also need `read` and/or `write` capabilities for the CDF resources you want to ingest data into. For instance, if you're ingesting **assets**, add `asset:read` and `asset:write`. If you revoke the capabilities in the CDF group, you also revoke access for the PostgreSQL gateway. Link the CDF group to a Microsoft Entra ID (ME-ID) group: 1. In the **Source ID** field, enter the **Object Id** for the ME-ID group exactly as it exists in ME-ID. 2. In the **Source name** field, enter the name of the group in Microsoft Entra ID. Select **Create**. ## Step 4: Create user credentials for the PostgreSQL gateway Send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/postgresgateway` [API](/api-reference/concepts/20230101/postgres-gateway-users). Where: * `cluster` is where your CDF instance exists. If you don't know the cluster name, contact [Cognite support](https://cognite.zendesk.com/hc/en-us/requests/new). * `project` is the organization name of your CDF instance. Get a **Bearer token** from your identity provider for authentication. To get a bearer token from Postman, see the illustration below. Get the sessions **nonce** using the Cognite API using the **Bearer token** obtained from above. **nonce** will expire after 1 minute, and must be used immediately. Provide the **nonce** to the API endpoint: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "credentials": { "nonce": "" } } ] } ``` The service returns, among other things, a **username** and **password** with the same CDF capabilities as the group you created in CDF. Make sure to keep these, as they are required when you set up the gateway connection in your ETL tool. Make sure you copy the **username** and **password**. If you lose the **password**, it can't be recovered. ## Migrate to PostgreSQL gateway 2.0 The PostgreSQL gateway 2.0 replaces the previous version (1.0). As indicated in [Step 4](#create-user-credentials-for-the-postgresql-gateway) above, this version uses `sessions` on behalf of your project and OIDC authentication to make requests to CDF. Most of the tables remain the same, but may have columns added or renamed. Some tables have been changed or divided up for clarity. These modifications are highlighted below: ### Assets Table name: *assets* **Added** | Columns | Type | | :- | :-: | | `created_time` | **TIMESTAMPTZ** | | `last_updated_time` | **TIMESTAMPTZ** | ### Event Table name: *events* **Added** | Columns | Type | | :- | :-: | | `created_time` | **TIMESTAMPTZ** | | `last_updated_time` | **TIMESTAMPTZ** | ### Extraction pipeline runs Table name: *extraction\_pipeline\_runs* **Added** | Columns | Type | | :- | :-: | | `id` | **BIGINT** | ### Extraction pipelines Table name: *extraction\_pipelines* **Added** | Columns | Type | | :- | :-: | | `contacts` | **JSON** | | `raw_tables` | **JSON** | | `documentation` | **VARCHAR(10000)** | | `last_updated_time` | **TIMESTAMPTZ** | | `created_by` | **VARCHAR(255)** | ### Files Table name: *files* **Added** | Columns | Type | | :- | :-: | | `directory` | **VARCHAR(512)** | | `asset_ids` | **BIGINT\[]** | | `source_created_time` | **TIMESTAMPTZ** | | `source_modified_time` | **TIMESTAMPTZ** | | `security_categories` | **BIGINT\[]** | | `created_time` | **TIMESTAMPTZ** | | `last_updated_time` | **TIMESTAMPTZ** | ### Labels Table name: *labels* **Added** | Columns | Type | | :- | :-: | | `data_set_id` | **BIGINT\[]** | | `created_time` | **TIMESTAMPTZ** | ### Sequences Table name: *sequences* **Added** | Columns | Type | | :- | :-: | | `created_time` | **TIMESTAMPTZ** | ### Sequence rows Table name: *sequence\_rows* **Changed** | Old | New | | :-: | :-: | | `row_values` ⇾ `values` | **JSON** ⇾ **JSON** | ### Datapoints The `datapoints` table has been divided into 2 different tables because version 1.0 only supported double datapoints. These are `string_datapoints` and `double_datapoints`. It includes the following columns: | Columns | Type | | :- | :-: | | `value` | **VARCHAR(255)** for `string_datapoints`
    **DOUBLE PRECISION** for `double_datapoints` | | `timestamp` | **TIMESTAMPTZ** | | `external_id` | **VARCHAR(255)** | | `id` | **BIGINT** | ### Timeseries The `timeseries` table has been renamed to `time_series`. **Removed** The `legacy_name` field has been retired and is no longer available for use. ### Raw The `raw` table is renamed to `raw_rows` for better understanding and proper differentiation between raw table data sets and raw table names. **raw\_rows** **Added** | Columns | Type | | :- | :-: | | `_raw_last_updated_time` | **TIMESTAMPTZ** | **Changed** | Columns | Type | | :- | :-: | | `database` ⇾ `_raw_database` | **VARCHAR** ⇾ **VARCHAR(32)** | | `table` ⇾ `_raw_table` | **VARCHAR** ⇾ **VARCHAR(64)** | | `key` ⇾ `_raw_key` | **VARCHAR(1024)** ⇾ **TEXT** | | `columns` ⇾ `_raw_columns` | **JSON** ⇾ **JSON** | **raw\_tables** **Added** | Columns | Type | | :- | :-: | | `_raw_database` | **VARCHAR(32)** | | `_raw_table` | **VARCHAR(64)** | # Ingest data into CDF RAW Source: https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw Use the PostgreSQL gateway to ingest data from ETL tools into CDF RAW tables. You need to define the table structure in the PostgreSQL gateway before setting up the sink/destination in your ETL tool. This documentation requires that you have completed [Step 3: Set PostgreSQL gateway as sink](/cdf/integration/guides/interfaces/postgres_gateway/setup_data_factory#set-postgresql-gateway-as-sink). ## Step 1: Manage tables in PostgreSQL gateway APIs The [Cognite API](/api-reference/concepts/20230101/api-description) will help create, retrieve, alter, and delete tables in the PostgreSQL gateway for a table in CDF RAW. ### Create a table Sign in to [Postman](https://www.getpostman.com). You can also use the web version of Postman if you haven't downloaded the tool. In your workspace, create a new request and enter the request name **Create table**. Select the request type as **POST** and enter your request URL - `https://{cluster}.cognitedata.com/api/v1/projects/{project}/postgresgateway/tables/{username}`. See [Setting up PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway#step-4-create-user-credentials-for-the-postgresql-gateway) for more information. Where: * `cluster` is where your CDF instance exists. If you don't know the cluster name, contact [Cognite support](https://cognite.zendesk.com/hc/en-us/requests/new). * `project` is the organization name of your CDF instance. * `username` with the username credentials On the **Authorization** tab, select **OAuth 2.0** as the type of Authorization. To get your access token, see [setting up PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway#step-4-create-user-credentials-for-the-postgresql-gateway). The **Body** tab allows you to specify the data you need to send with a request for every POST request. Inside the Body tab, select **raw** as the data model and select **JSON** from the dropdown as the data format and enter the below JSON: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "type": "raw_rows", "tablename": "tablename", "options": { "database": "my_database", /* my_database is your database name in CDF */ "table": "my_table", /* my_table is your table name in CDF */ "primaryKey": "title" }, "columns": { "title": {"type": "TEXT"}, "year": {"type":"TIMESTAMPTZ"} } } ] } ``` `tablename` cannot be any of these built-in table names: assets, double\_datapoints, events, extraction\_pipeline\_runs, extraction\_pipelines, files, labels, raw\_rows, raw\_tables, relationships, sequence\_rows, sequences, string\_datapoints, time\_series, data\_models, views, containers, units, unit\_systems Select **Send** to fetch a response. Postman interface showing successful table creation You have successfully created a new table. When you've created a table in the PostgreSQL gateway, the corresponding table in CDF RAW is created when the first row is ingested using the PostgreSQL gateway. ### List tables Create a new request and enter the request name **List tables**. Select the request type as **GET** and enter your request URL as `https://{cluster}.cognitedata.com/api/v1/projects/{project}/postgresgateway/tables/{username}` Select Authorization type as **OAuth2.0**. Select **Send** to fetch the response. Postman showing list of custom tables ### Delete a table Create a new request and enter the request name **Delete table**. Select the request type as **POST** and enter your request URL as `https://{cluster}.cognitedata.com/api/v1/projects/{project}/postgresgateway/tables/{username}/delete` Select Authorization type as **OAuth2.0**. Inside the **Body** tab, select **raw** as the data model and select **JSON** as the data format. Enter this JSON: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [{ "tablename": "" }] } ``` Only custom tables created with the examples above can be deleted. Select **Send** to fetch the response. Postman showing successful table deletion `{}` in the response indicates you have successfully deleted the table. When you delete a table, it's also deleted in the PostgreSQL gateway, but not from CDF RAW. ## Step 2: Complete the connection in ETL Verify that the table you created above is available as a foreign table in your ETL tool. Configure your ETL pipeline between the source system and CDF RAW. # Use Azure Data Factory with PostgreSQL gateway Source: https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway/setup_data_factory Create data pipelines in Azure Data Factory to ingest data into Cognite Data Fusion (CDF) through the PostgreSQL gateway. Ingest data with your existing ETL tool into the CDF staging area, [RAW](/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw), or into the CDF data model with the PostgreSQL gateway. This article shows how to create a **simple** data pipeline in ADF to copy data from a file in Azure blob storage and ingest it to CDF. Based on your data source and your data processing requirements you may need to create more advanced data pipelines in [Azure Data Factory](https://docs.microsoft.com/en-in/azure/data-factory/data-factory-tutorials). ## Before you start * Make sure you have an Azure subscription and have created [Azure resource groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal). * Make sure you have created an [Azure storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create). * Make sure you have the file containing the data to be ingested to CDF in Azure blob storage. ## Step 1: Create a data factory 1. Sign in to the [Azure portal](https://portal.azure.com) as an admin. 2. If you have access to multiple tenants, use the Directory + subscription filter in the top menu to select the tenant in which you want to register a data factory. Search for and select **Data Factory**. Select **Create** to open the **Data Factory** page. Azure portal showing Data Factory creation page On the **Basics** tab: * **Subscription:** Select the Azure subscription in which you want to create the data factory. * **Resource group:** Create or select the Azure resource group you want to use. * **Region:** Select a location where your ADF metadata will be stored. This list only shows locations that ADF supports. * **Name:** Enter a globally unique name for the data factory. * **Version:** Select V2. Select **Next: Git configuration** and fill in the required fields or select **Configure Git later**. Select **Review + Create** > **Create**. After the creation is complete, open the data factory and select the **Author & Monitor** tile to start the Azure Data Factory application in a separate tab. ## Step 2: Create an ADF pipeline and set the source system Use the **Copy data** tool to create a pipeline that reads data from a file in your data storage and writes to CDF. On the **Let's get started** page, select the **Create pipeline** tile. Open **Activities** > **Move and transform**. Drag and drop the **Copy data** onto the canvas and enter a unique name in the **Name** field. Open the **Source** tab > **New**. Select the **datastore** that references the source system you're extracting from. In this example, we'll extract data from a .csv file in Azure Blob Storage. Select **Continue**. Azure Data Factory copy data configuration Create a **new linked service** that links your data store to the data factory. 1. Enter a unique **name** for the service. 2. Select your **Azure subscription** and **storage account**. 3. Select **Test connection** to make sure you are connected to the selected storage account. 4. Select **Create**. Azure Data Factory source configuration showing linked service setup Under **Set properties**, set the file you want to read data from. 1. Enter a unique **name** for the source data set. 2. Browse to the file you want to use. 3. Select **First row as header** to reuse the source file's header columns. 4. Select **OK**. ## Step 3: Set PostgreSQL gateway as sink In this step, add the PostgreSQL gateway as the destination using a **sink** in ADF. Open **Copy data** > **Sink** tab. In the **Sink dataset** field, select **New**. Select or search for **Azure Database for PostgreSQL**. Select **Continue**. Select **Open** to open the sink data set. Azure Data Factory sink dataset configuration On the **Connection** tab, add a linked service for the sink data set. 1. Select **New** to add a linked service. 2. Under **Account selection method** > **Enter manually**. 3. In the **Fully qualified domain name** field, enter `fdw.{cluster}.cogniteapp.com` where `{cluster}` is where your CDF instance is installed. If you don't know the cluster name, contact [Cognite support](https://cognite.zendesk.com/hc/en-us/requests/new). 4. In the **Port** field, enter 5432, which is the default port for the PostgreSQL gateway. 5. For **Database name** and **User name**, enter the [username returned from the PostgreSQL gateway](/cdf/integration/guides/interfaces/postgres_gateway/postgres_gateway_basics). 6. In the **Password** field, enter the password returned from the PostgreSQL gateway. 7. In the **Encryption method** field, select **SSL**. This ensures that the communication between ADF and CDF are encrypted. 8. Optionally, select **Validate server certificate**. 9. Select **Test connection** to make sure you are connected to the Azure database for PostgreSQL. Select **Create**. ## Step 4: Set the destination database table Back on the **Connection** tab, you need to select the destination table from the tables available in your PostgreSQL gateway database. You can choose between several resource types like assets, events, and data points. In the **Table** field, select the table matching the destination [CDF RAW table](/cdf/integration/guides/interfaces/postgres_gateway/ingest_into_raw) or the CDF resource type `public.`, for instance, *public.assets* or *public.time\_series*. Azure Data Factory showing available public tables ## Step 5: Map the source and destination tables Open the **Mapping** tab on your pipeline. Select **Import schemas** to import the table columns and remove the lines you won't use. Map the columns according to your requirements. Select **Debug** in the top bar to run the PostgreSQL gateway. Azure Data Factory mapping configuration between source and destination columns ## Step 6: Report run status for data extractions Monitor the data integration into CDF to quickly catch failed runs to start troubleshooting and notify data set owners and stakeholders. You can see an overview of all extraction pipelines' status on the [Extraction pipelines](/cdf/integration/guides/interfaces/about_integrations) page. To set up status reporting: Open Azure Data Factory. Navigate to the **Copy data** activity created for the extraction pipeline, select the plus (+) sign, and select **Success**. Azure Data Factory showing success status reporting configuration Drag and drop an additional **Copy data** activity onto the canvas and link it to the **Success** activity. On the **Source** tab: * Enter a source data set, but note that no data is read from this data set. You can, for instance, enter the data set already used in the previous pipeline. * Create two additional columns named `integration_status` with content `success` and `integration_external_id` with the [extraction pipeline's](/cdf/integration/guides/interfaces/about_integrations) external ID as content. Azure Data Factory additional columns configuration for status reporting On the **Sink** tab: 1. In the **Sink dataset** field, select **New** and reuse the sink and linked service from the previous pipeline. 2. In the **Table** field, browse and select the foreign table `public.extraction_pipeline_runs`. Azure Data Factory extraction pipeline run status table configuration For a large dataset, set the `Write method` to `Copy command`. On the **Mapping** tab, create two mapping rows: * Row 1: Enter `integration_external_ID` as **source** and the `externalId` as **destination**. * Row 2: Enter `integration_status` as **source** and `status` as **destination**. Navigate back to the **Copy data** activity created for the pipeline, select the plus (+) sign, and select **Failure**. Drag and drop an additional **Copy data** activity onto the canvas and link to the **Failure** activity. Follow the steps above, but on the **Source** tab, enter `failure` as content for the `integration_status`column. Azure Data Factory complete status reporting setup showing success and failure paths ## Troubleshooting **Invalid token** If you have a valid token but still get `invalid token` when you create a user in the PostgreSQL gateway, try these solutions: * The groups in Microsoft Entra ID and CDF are correctly linked in **Manage & Configure** > **Access management** in CDF. See step 3 above. * The CDF group is set up with the `Project:list` and `Group:list` capabilities in **Manage & Configure** > **Access management** in CDF. # SAP writeback service Source: https://docs.cognite.com/cdf/integration/guides/interfaces/sap_writeback Learn about the SAP writeback service, including CogniteFile attachment references for data modeling projects. The SAP writeback service sends data from Cognite Data Fusion (CDF) to SAP S/4HANA through configured OData endpoints. Use it to create and update SAP records, including attachments, from CDF. ## Supported SAP entities The SAP writeback service connects to SAP standard delivered OData V2.0 endpoints available in SAP S/4HANA. It supports the following SAP entities: * [Maintenance notifications](https://api.sap.com/api/OP_API_MAINTNOTIFICATION/overview) * [Attachments](https://api.sap.com/api/OP_API_CV_ATTACHMENT_SRV_0001/overview) Attachment writeback requests include a file reference on each request item. Choose the reference type based on how the file is stored in CDF. | How the file is stored | File reference | | - | - | | Data modeling (`CogniteFile`) | `fileReference` with `type: "cdm_file"`, plus the `space` and `externalId` of the instance | | Asset-centric (legacy) | Legacy `fileId` (file external ID), or `fileReference` with `type: "cdf_classic_file"` | In the API, `type: "cdf_classic_file"` is synonymous with an [asset-centric (legacy)](/api-reference/concepts/20230101/files) file. Use asset-centric file references in asset-centric and hybrid projects. Data modeling projects must use `cdm_file` with a `CogniteFile` instance. Don't set both `fileId` and `fileReference` on the same request item. The request fails with HTTP 422. Applications such as Cognite InField can send attachment writeback using the same `fileReference` fields. To send an attachment writeback request, see [Set up the SAP writeback service](/cdf/integration/guides/interfaces/sap_writeback_setup#send-an-attachment-writeback-request). For the full request schema, see [Writeback requests](/api-reference/concepts/20230101/writeback-requests). ## Supported SAP versions The SAP writeback service supports the following SAP S/4HANA versions: * SAP S/4HANA OnPremise 2021 FPS01, and later * SAP S/4HANA Cloud ## Before you start * Make sure you [register the Cognite API and applications in Microsoft Entra ID](/cdf/access/entra/guides/configure_cdf_azure_oidc) and [set up Microsoft Entra ID and CDF groups](/cdf/access/entra/guides/create_groups_oidc) to control access to CDF data. * Assign the [SAP writeback capabilities](/cdf/access/guides/capabilities#sap-writeback) in a Cognite Data Fusion (CDF) project. * Make sure the SAP destination endpoints are [active](https://help.sap.com/docs/SAP_Group_Reporting_Data_Collection/e29950571a2b4065b1dacaf881c67413/92bce22bebe146db826a22e64f7b7acd.html) and accessible from outside your intranet. * Make sure the SAP user has [authorization](https://help.sap.com/doc/saphelp_ssb/1.0/en-US/6f/0e415370107d77e10000000a441470/content.htm?no_cache=true) to call SAP OData services. # Set up the SAP writeback service Source: https://docs.cognite.com/cdf/integration/guides/interfaces/sap_writeback_setup Step-by-step guide to set up the SAP writeback service and send notification and attachment requests to SAP S/4HANA from Cognite Data Fusion (CDF). Follow these steps to configure the SAP writeback service and send writeback requests to SAP S/4HANA. ## Before you start Complete the prerequisites in [SAP writeback service](/cdf/integration/guides/interfaces/sap_writeback#before-you-start). To send requests to your SAP S/4HANA instance, set up the SAP host destination. An SAP instance is a configuration for an external SAP S/4HANA destination system. The instance resource contains all the information the SAP writeback service needs to connect to the SAP S/4HANA destination. Send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/instances` to create an SAP instance destination. See [API documentation](/api-reference/concepts/20230101/sap-instances) for more information. The SAP writeback service provides schema mapping functionality to map a CDF resource data schema to an SAP entity schema. Define the mapping with the Cognite extractor [mapping language](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts). If you set a mapping, the SAP writeback service will apply the mapping expression before sending the request to the SAP destination. The example below contains an expression mapping the Cognite InField observation schema to the SAP maintenance notification schema: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "NotificationText": substring(input.description,0,40), "MaintPriority": case(input.priority,"Very High","1","Medium","2","Low","3"), "NotificationType": input.type, "MaintNotifLongTextForEdit":concat(input.description,input.troubleshooting), } ``` Write the mapping expressions to fulfill your business needs regarding field or value mapping between Cognite Data Fusion and SAP entities. To save your mapping, send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/mappings`. See [API documentation](/api-reference/concepts/20230101/sap-schema-mappings) for more information. In the SAP writeback service, an SAP **endpoint** is a configuration to an SAP S/4HANA OData endpoint (and its related OData entity). The API will send writeback requests to this endpoint. It defines which SAP instance destination and schema mapping should be used when processing a writeback request. Use the SAP **endpoint** as a reference to send requests to your SAP destination. To set the SAP **endpoint**, send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/endpoints`. See [API documentation](/api-reference/concepts/20230101/sap-endpoints) for more information. The SAP writeback service provides an endpoint that checks if the connectivity between CDF and the configured SAP **endpoint** destination is successful. Send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/endpoints/verify`. See [API documentation](/api-reference/concepts/20230101/sap-endpoints) for more information. For a successful connection, the response from the SAP writeback `/verify` endpoint looks like: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "status": "success" } ``` Once you're done with the setup, send requests to your SAP destination. The SAP writeback service sends all requests to SAP asynchronously. Once you send a request through the service, you'll get a `requestId`. You can use the `requestId` to track the `status` of the request once it is completed. Send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/requests` with the payload to be sent to SAP. See [API documentation](/api-reference/concepts/20230101/writeback-requests) for more information. This example shows a SAP writeback request to create a SAP maintenance notification. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "endpointId": "test-notification", "request": [ { "payload": { "MaintNotifLongTextForEdit": "test description", "MaintPriority": "3", "NotificationText": "Test", "NotificationType": "M2" } } ] } ] } ``` Response: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "createdTime": 1724416805132, "lastUpdatedTime": 1724416805132, "requestId": "0a0a0-1b1b-2c2c-3d3d-4e4e4e", "status": "pending", "request": [ { "payload": { "MaintPriority": "3", "NotificationText": "Test", "NotificationType": "M2", "MaintNotifLongTextForEdit": "test description" } } ] } ] } ``` To check the request status, send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/requests/byids` with the related `requestId`. See [API documentation](/api-reference/concepts/20230101/writeback-requests) for more information. Request: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items":[ { "requestId":"0a0a0-1b1b-2c2c-3d3d-4e4e4e" } ] } ``` Response: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "createdTime": 1724416805132, "lastUpdatedTime": 1724416806161, "requestId": "0a0a0-1b1b-2c2c-3d3d-4e4e4e", "status": "done", "request": [ { "sapObjectId": "10129745", "payload": { "MaintPriority": "3", "NotificationText": "Test", "NotificationType": "M2", "MaintNotifLongTextForEdit": "test description" } } ] } ] } ``` The writeback request is completed when the request status is set to `done`. The response parameter `sapObjectId` contains the object ID of the record created in the SAP destination. Sign in to your SAP server and go to transaction code `IW23` to check the **maintenance notification** created by the writeback request. Input the `sapObjectId` value available in the writeback response. SAP maintenance notification showing created record To attach a file to an SAP object, use an SAP endpoint with `endpointType: attachment`. See [SAP endpoints](/api-reference/concepts/20230101/sap-endpoints) for more information. Before you send the request, make sure the file exists in CDF in **one** of these forms: * **Data modeling:** a `CogniteFile` instance — see [Files in data modeling](/cdf/dm/dm_guides/dm_integrate_files) * **Asset-centric (legacy):** a file created with the [Files API](/api-reference/concepts/20230101/files). In the API, `type: "cdf_classic_file"` is synonymous with an asset-centric (legacy) file. Use these references in asset-centric and hybrid projects only. For data modeling projects, use `fileReference` with `type: "cdm_file"` and the `CogniteFile` `space` and `externalId`. Data modeling projects can't use `fileId` or `type: "cdf_classic_file"`. In the request payload, set `sourceId` to the SAP notification ID that the attachment belongs to. Also provide either `fileId` or `fileReference` — not both. When you use `fileId`, set it to the file external ID. ### Reference a CogniteFile Send a **POST request** to `https://{cluster}.cognitedata.com/api/v1/projects/{project}/writeback/sap/requests`. This example references a `CogniteFile` instance: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "endpointId": "my-attachment-endpoint", "request": [ { "payload": { "sourceId": "123456" }, "fileReference": { "type": "cdm_file", "space": "my_dm_space", "externalId": "my-file-external-id" } } ] } ] } ``` ### Reference an asset-centric (legacy) file In asset-centric and hybrid projects, use either legacy `fileId` (file external ID) or `fileReference` with `type: "cdf_classic_file"`: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "endpointId": "my-attachment-endpoint", "request": [ { "payload": { "sourceId": "123456" }, "fileId": "my-file-external-id" } ] } ] } ``` ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "endpointId": "my-attachment-endpoint", "request": [ { "payload": { "sourceId": "123456" }, "fileReference": { "type": "cdf_classic_file", "externalId": "my-file-external-id" } } ] } ] } ``` The service fetches file content through the [File content](/api-reference/concepts/20230101/file-content) API (`/files/downloadlink`) for both asset-centric (legacy) files and `CogniteFile` instances. The SAP attachment OData endpoint is unchanged. Poll the request status with `requests/byids`, as in the [Send a request](/cdf/integration/guides/interfaces/sap_writeback_setup#send-a-request) step. Don't set both `fileId` and `fileReference` on the same request item. The request fails with HTTP 422. For the full request schema, see [Writeback requests](/api-reference/concepts/20230101/writeback-requests). The writeback service can send update requests to SAP. The request is similar to the `create` request in the [Send a request](/cdf/integration/guides/interfaces/sap_writeback_setup#send-a-request) step. However, the object ID of the SAP record must be available in the request payload. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "items": [ { "endpointId": "test-notification", "request": [ { "payload": { "MaintenanceNotification":"10129745", //Object ID of the SAP Maintenance Notification you want to update "MaintNotifLongTextForEdit": "Updated description", } } ] } ] } ``` # Set up on-premises integrations Source: https://docs.cognite.com/cdf/integration/guides/interfaces/set_up_on_premises_integration Create on-premises integrations in Cognite Data Fusion (CDF), install the extractor, and complete the connection. Register an on-premises extractor as an **integration**, install it, and complete the **connection** to monitor **tasks**. Integrations is in [public preview](/cdf/product_feature_status#public-preview) and may change. Creating an integration creates the integration resource only. It does **not** create an extraction pipeline. Do not use the Integrations path and the Extraction pipelines path for the same external ID at the same time. ## Before you start * Confirm you can open Data fusion > Integrate > Integrations. If not, contact [Cognite Support](/cdf/support) or your Cognite representative. * Assign Integrations and destination [capabilities](/cdf/access/guides/capabilities#integrations) for the extractor. * Review the setup guide for your source: [DB](/cdf/integration/guides/extraction/db/db_setup), [File](/cdf/integration/guides/extraction/file/file_setup), [SharePoint Online](/cdf/integration/guides/extraction/file/file_sharepoint_setup), or [OPC UA](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup). For on-premises integrations, assign `integrations:read`, `integrations:write`, `integrations:use`, `integrationconfigs:read`, and `integrationconfigs:write`. You do **not** need `extractionpipelines:*`, `extractionruns:*`, or `extractionconfigs:*` for this path. ## Create the integration Navigate to Data fusion > Integrate > Integrations. Select Set up. CDF opens Extractors so you can choose the extractor. Select a supported on-premises extractor. For SharePoint Online, use the File extractor. Enter the name, external ID, and other required fields, then save. CDF creates the **integration** and shows download and connection instructions. You can also open Extractors under Integrate at any time. ## Complete the connection Follow the on-page instructions. When the connection succeeds, the extractor has finished startup and check-in with CDF. **Tasks** appear only after that so you can monitor them. Download the package for your environment. Install the extractor on a host that can reach the source system and CDF. Add CDF authentication to the local extractor configuration, then start the extractor so it can check in with CDF. Keep secrets out of shared samples. Open the integration, select the Configuration tab, paste your local configuration file content, validate, and publish. Configuration is keyed by the integration external ID. You can also download packages from Data fusion > Integrate > Extractors. See [About data extraction](/cdf/integration/concepts/extraction#download-installation-files). When the connection succeeds, open Overview tab and confirm that **tasks** are listed. ## Next steps * Complete source-specific setup in the [DB](/cdf/integration/guides/extraction/db/db_setup), [File](/cdf/integration/guides/extraction/file/file_setup), or [OPC UA](/cdf/integration/guides/extraction/opc_ua/opc_ua_setup) guide. * Review [Configuration settings](/cdf/integration/guides/interfaces/configuration_settings) for connection and application configuration. * [Monitor integrations](/cdf/integration/guides/interfaces/monitor_unified_integrations). * To keep using Extraction pipelines, see [Set up extraction pipelines](/cdf/integration/guides/interfaces/add_integrations). # Contextualize 360° images Source: https://docs.cognite.com/cdf/3d/guides/360images_contextualize Manually contextualize 360° images by using a single 360° image file or leveraging the 360° image contextualization within a scene. When **contextualizing 3D objects** (CAD models, point clouds, and 360° images), you can use the unified 3D contextualization editor available in [scene](/cdf/3d/guides/scene#unified-3d-contextualization). By selecting this option, you'll see where each 360° image is located in the scene, which helps you link the images to the right assets. ## Before you start Make sure you've uploaded the 360° images you want to contextualize. For more information on uploading 360° images, see [Upload 360° images](/cdf/3d/guides/360images_upload). ### Trigger text detection (data modeling projects) Before contextualizing, you can **trigger text detection** to create contextualization suggestions for text regions that you can link to the correct asset to accelerate contextualization. This feature is available in **data modeling** projects only. In **Data fusion** > **Configure** > **3D**, select the 360° image collection you want to process. Select the `...` menu in the lower right of the collection view and select Trigger text detection. Processing runs on the images in the collection and generates contextualization suggestions. After it has been triggered, the option is grayed out because it cannot be triggered multiple times. After text detection completes, open the 360° image collection to see the contextualization suggestions (yellow areas). To link each suggestion to an asset, follow **Find the asset** and **Link the asset** steps in [Contextualize](#contextualize). ## Contextualize Select the 360° image collection and **Open** it to start the contextualization process within the 3D environment. **Contextualize 360° images within a scene** From the scene tree view, select the 360° image collection you want to contextualize. Select Contextualize to start. For more information, see [Unified 3D contextualization](/cdf/3d/guides/scene#unified-3d-contextualization). Select a 360° image to open it. Inside the 360° image, you'll have the contextualization tools. Select to create a bounding area in a 360° image. Select a point and move the cursor to create a box of any shape. Press Esc to complete the drawing. You can create multiple bounding areas. By default, the color of the area is blue, but it switches to pink once you link it to an asset. Use the search bar in the resource selector on the top right to find the asset you want to map to the selected object. Select the asset, verify it is correct, and select **Link**. You've created an annotation (contextualized bounding area). The newly created annotation appears under the selected image in the left sidebar. You can see which 360° images have annotations. ### Tools in contextualization * In the top bar, you can navigate between 360° images. X closes the image, and you can return to the last one you opened. * Settings opens visualization controls: * Image transparency: Use the slider to control the transparency level of the 360° image with the CAD model in the background. This fine-tunes the visibility between the two layers. * Marker transparency: Adjust marker transparency to balance visibility with other visual elements. * 360° markers: Customize marker visibility to enhance your image analysis and inspection experience: * X-ray: Reveal the hidden markers or display all the 360° markers within the same view. * Visible: Turn markers on and off. * Turn on and off grid axis directions using Show or hide axis in the toolbar. * To zoom an annotation, select Zoom to annotation. ## Delete annotation To delete the annotation: Select the icon from the toolbar and select the annotated area. Select in the top right corner. You've deleted the annotation. # Upload 360° images Source: https://docs.cognite.com/cdf/3d/guides/360images_upload You can upload 360° images to Cognite Data Fusion (CDF) and display them in various applications with the Reveal 3D viewer. 360° images can be captured as part of a point cloud scan, such as an E57 format file, but you can also capture and upload them manually or using robots. In Reveal 3D viewer, 360° image locations appear as **round markers**. Select a marker to enter the 360° image, and use your mouse to rotate the camera in any direction. In the 360° image, select round markers on the floor to move between images, or click to move to the nearest station in that direction. If the scene includes a reference point cloud model, a **directional cursor** shows where you're pointing. Smooth transitions help you keep orientation when moving between stations, and an elevation indicator helps you distinguish stations on different floor levels. 360 degree markers displayed in Reveal 3D viewer You need to **upload** the image files to CDF. Each station has six **cube map images** with a specific set of **metadata**. These cube map images together form a 360° image, representing six directions (left, front, right, back, top, and bottom). You can access the cube map images separately through file exports from the lidar scanning software or in E57 point cloud files. Six cube map images forming a 360 degree image ## Automatic conversion for E57 files E57 files can contain 360° images in various formats, including **cube maps** and **equirectangular** (spherical) projections. If a 360° image is in a format other than a cube map, the extraction process will try to convert it to the cube map format. This conversion may require additional processing time and result in a slight quality loss compared to images that are already in cube map format. See [E57 files with 360° images](#e57-files-with-360-images) for upload instructions. ## Manual conversion for other file formats If you have equirectangular images in other file formats (not E57), convert them to cube maps before uploading to CDF: 1. Use a conversion tool such as [py360convert](https://github.com/sunset1995/py360convert) to convert your images to cube map format. 2. Upload the converted cube maps following the steps in [Upload 360° images programmatically](/dev/3d/api_asset_centric/upload_360#custom-script-for-other-formatted-360-images). ## Before you start ### Access capabilities See [access capabilities](/cdf/access/guides/capabilities#upload-3d-models) to add the necessary capabilities to upload and view 360° images. ### Limitations 3D API processing now supports extracting 360° images from E57 files for a 3D model with a `data set` when starting from a script. The CDF user interface currently doesn't support triggering the extraction of 360° images when either the source `fileId` or `modelId` has a data set. The extracted 360° image files will be stored in the same data set as the 3D model. If the 3D model doesn't have a `data set`, the 3D API will also check and use any data set associated with the source file specified for the revision. If you want the extracted 360° images to be available in a different data set, change the `data set` association for the files after processing is complete using file update requests. If a data set is used in a `modelId` or source `fileId`, specify a `dmsInstancesSpaceId` value in the *indexingOptions* for the revision's request body. See [Upload 360° images programmatically](/dev/3d/api_asset_centric/upload_360#trigger-360-image-extraction-from-e57-files-using-python) for how to trigger extraction from a script. This specifies the `space` where the data model instances for 360° images are stored. Uploading the 360° image metadata (pose information) to the [Events API](/api-reference/concepts/20230101/events) is no longer possible. This functionality has been deprecated and will stop working in the future. ## Get started You can upload 360° images to CDF in two ways: * [By uploading E57 files with 360° images](/cdf/3d/guides/360images_upload#e57-files-with-360-images), if you have a point cloud file in the E57 file format that includes both 360° images and point cloud data. A zip file of E57 files is also supported. This is only supported for the asset-centric 3D models. * [By uploading other formatted 360° images](/dev/3d/api_asset_centric/upload_360#custom-script-for-other-formatted-360-images), if you have 360° images and related custom metadata in separate, other (not E57) formatted files. After uploading in the user interface or using a custom script, you can view 360° images in Search by adding the 360° image collection to a [scene](#add-360-images-to-the-scene). To view 360° images in custom Reveal applications, see [Reveal documentation](https://cognitedata.github.io/reveal-docs/docs). ## E57 files with 360° images Follow the steps to upload the E57 files to CDF, trigger point cloud filtering and indexing, and opt-in to 360° image extraction. If these files are large: * Upload the files in the ZIP format using the [file extractor](/cdf/3d/guides/3d_file_extractor_guide). * Or, upload them as individual E57 files, referenced through a JSON file using [a custom Python script](/dev/3d/api_asset_centric/upload_pointcloud). Individual E57, ZIP, or JSON. Navigate to **Data fusion** > **Configure** > **3D** > **Upload model**. Enter a model name, leave the data set field empty, and then select Upload new model. Upload the file, if needed, or select *Choose existing uploaded file to process to create a 3D model* and enter the file ID (from step 2). Select **Upload and process**. For smaller files, this can take a few minutes. For larger files, this might take several hours or days. Refresh the page if necessary. 1. Once the model's status changes to *Done*, select the model > ... > Filtering and indexing. Filtering and indexing 2. Select Use the file ID from the previous revision. 3. Select Extract 360° images and clear Generate tiles option. 4. **Optional**: Expand the Filtering section and manage more settings. 5. Select **OK** to start the point cloud processing. You can view the processed point cloud model once the processing is complete. To trigger 360° image extraction from E57 files with a script, or to upload and register other 360° formats (for example, cube maps) via the API, see [Upload 360° images programmatically](/dev/3d/api_asset_centric/upload_360). ## Add 360° images to the scene To see the 360° images in CDF programs and applications using the Reveal 3D viewer, [add 360° images to the scene](/cdf/3d/guides/scene). For the E57 file (set) with both point cloud and 360° images, add 3D data when you configure or edit the scene. See [Unified 3D contextualization](/cdf/3d/guides/scene#unified-3d-contextualization) for more information. The 360° images extracted from the E57 file set will be put in a 360° image collection named `site--`. For example, `site-111-222`, where 111 is the model ID and 222 is the revision ID. # Upload 3D models with file extractor Source: https://docs.cognite.com/cdf/3d/guides/3d_file_extractor_guide From **Upload 3D models** in Cognite Data Fusion (CDF), you can upload 3D models and revisions to CDF without writing any code. CDF supports CAD files from common systems and standard formats such as RVM, OBJ, and FBX. For more information on supported formats, see [Supported 3D file formats](/cdf/3d/guides/3dmodels_intro#supported-3d-file-formats). ## Get started To upload 3D models, use the **file extractor** in the CDF. The file extractor is a tool that extracts 3D models from CAD files and uploads them to CDF. The file extractor is available in the CDF under **Integrate** > **Connect to source systems**. See the [Cognite File Extractor](/cdf/integration/guides/extraction/file/file_setup). Once you set up the file extractor, you can upload 3D models to CDF by following the extractor's instructions. Once the upload is complete, find the **file-id** of the uploaded 3D model and use it to create a 3D model or revision for an existing 3D model in CDF. Identify the **file-id** of the uploaded 3D model by enabling console logs in the file extractor and examining the logs for the **file-id**. Adjust the log level to DEBUG in the file extractor configuration file and subsequently run the file extractor. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} logger: # Logging to console/terminal. Remove or comment out to turn off the terminal # logging console: level: DEBUG ``` Console log showing file ID from file extractor Locate the **file-id** of the uploaded 3D model. Navigate to the [CDF homepage](https://fusion.cognite.com) > **Data fusion** > **Data explorer**. Next, select **Files**, and find the **file-id** of the uploaded 3D model by searching for the file name. Finding file ID in CDF Data Explorer ## Process a 3D model Navigate to **Data fusion** > **Configure** > **3D**. Select Upload model and follow the wizard to create the model and upload the initial revision of the model. Upload 3D model revision dialog ## More information To upload a revision to an existing 3D model, see [Upload a revision](/cdf/3d/guides/3dmodels_upload#upload-a-revision-to-an-existing-3d-model). # Configure CDF 3D Visual in Power BI Source: https://docs.cognite.com/cdf/3d/guides/3dmodels_PowerBI Display and interact with 3D CAD models from Cognite Data Fusion (CDF) directly in Power BI reports. The **Cognite Data Fusion 3D Visual** uses Cognite's [Reveal 3D](https://cognitedata.github.io/reveal-docs/docs) library to embed industrial asset models in your dashboards. To use this visual, your 3D models must be stored in CDF using Cognite's **asset-centric data models**. Models stored in CDF core data models are not yet supported. You can navigate and select model components to connect your Power BI data with physical asset locations for enhanced spatial analysis. Power BI model viewer ## Before you start Make sure you have: * Access to a **CDF project** with permissions to view 3D models * An active **CDF license** that allows token acquisition * [Power BI Desktop](https://powerbi.microsoft.com/desktop/) installed. ## Step 1: Configure Microsoft Entra ID permissions This allows the Power BI visual to securely connect to your CDF project. This step requires administrator permissions and is a one-time setup per tenant. * Replace **TENANTID** with your tenant ID (found in Azure Portal > Microsoft Entra ID > Overview) * Replace **CLUSTER** with your Cognite cluster name First URL: ``` https://login.microsoftonline.com//adminconsent?client_id=https://powerbivisual.cognitedata.com ``` Second URL: ``` https://powerbivisual.cogniteapp.com/consent?tenantid=&cluster= ``` Open both URLs in your browser and sign in with an admin account to grant permissions. ## Step 2: Add data to your report Prepare a data set that you'll connect to your Power BI report. It should include the following 3D model information: * **Model name** (string) — the name of the 3D model * **Model ID** (number) — the unique identifier for the model in CDF * **Revision ID** (number) — the revision identifier for the model Select **Get data** from the **Home** ribbon in Power BI to connect to data. Once you load the data to the report, it's visible in the **Data** pane. ## Step 3: Add the 3D visual to your report In your Power BI report, select **More options (...)** in the **Visualizations** pane. Import the visual from a local *.pbiviz* file. 1. Select **Import a visual from a file**. 2. Browse to your file location and select it. 3. Confirm the import when prompted. Download the visual directly from AppSource. 1. Select **Get more visuals**. 2. Search for **Cognite Data Fusion 3D Visual** in the AppSource marketplace. 3. Add the visual and confirm the import. Select the new visual icon in the **Visualizations** pane to use this visual in the report. ## Step 4: Authenticate with CDF Connect the visual to your CDF project. Select **Get Started** on the landing page. In the authentication dialog, enter your **CDF project name** and **cluster name**. Select **OK** to authenticate. If authentication fails, verify that your CDF project name and cluster name are correct, and that your access token is valid. ## Step 5: Bind the data From the **Data** pane, drag **Model name**, **Model ID**, and **Revision ID** to the visual's **Filters** section. If your data set doesn't have these fields, or they are incomplete, you can browse and select models manually using the **Show model list** icon in the 3D toolbar. After binding the data, verify that the visual loads the specified 3D model in the viewer. ## Step 6: Interact with the 3D viewer Once a 3D model is loaded, you can navigate and interact with it using the viewer's toolbar. **Toolbar features** * Filter the list of available 3D models * Analyze and measure objects * View node properties of the selected CAD model components ## Troubleshooting Find information to help you troubleshoot common issues. If the authentication fails or you cannot acquire a token: Verify your CDF license includes token acquisition permissions. Check if your access token has expired: 1. Sign out of Power BI. 2. Sign back in to generate a new token. If the visual displays **No data available**: Check that your data set includes **Model ID** and **Revision ID** columns. Ensure the data is bound to the visual. If you see a server error message: Verify the CDF cluster is operational. 1. Select **Get Started** in the visual. 2. Enter your credentials again. ## More information * [About 3D models](./3dmodels_intro) * [Upload 3D models in the browser](./3dmodels_upload) * [Reveal 3D Web Viewer SDK](https://cognitedata.github.io/reveal-docs/docs) # Contextualize CAD models Source: https://docs.cognite.com/cdf/3d/guides/3dmodels_contextualize With the manual contextualization of CAD models, you can map assets to 3D objects, improving the assets discoverability when using search. ## Before you start Before contextualizing, make sure you have already uploaded the CAD model. If you haven't uploaded yet: * Navigate to **Data fusion** > **Configure** > **3D** to upload your CAD models. * Select Upload model and follow the wizard to upload a model. For more information on uploading CAD models, see [Upload 3D models](/cdf/3d/guides/3dmodels_upload). ## Get started Find the CAD model you want to contextualize from the list of uploaded models. Select **Open**. Within the 3D environment: * Choose the objects by selecting them directly. * Alternatively, select the system or objects you want to contextualize by navigating the *3D node hierarchy*. Use the search bar on the right to search for an existing asset and scroll through the list. Select the asset, verify it is correct, and select **+ Add** at the top right to add the asset. Use the *3D node hierarchy* to view the linked asset. Select the linked asset to highlight it in the 3D environment. Select **Create annotation**. If you want to map additional assets to the same node selection, repeat steps 2-4 for each asset. To map an asset to a selection that includes already selected objects, add the additional objects to the existing selections. Once you have finished mapping assets to the selection, check your selection to avoid ending up with incorrectly contextualized objects. ## Validate mappings To validate and check your mapping, select **Annotations** in the top left corner of the viewer. This will collapse a list of all the mapped assets in the model. Hover over the list item to highlight the mapped objects. Select a list item to reposition the camera view to display all linked objects to the asset. To open the 3D model in Data Explorer, select `...` in the top right corner and select **Open in Data Explorer**. Select in the left corner of the toolbar to highlight the contextualized objects. Now, when you select an object, the detail card for the asset will appear. In CAD, you can map several assets to one object. ## Delete links If you incorrectly map one or more objects to an asset, you can delete this link by selecting **Annotations** in the top left corner. This will collapse a list of all the mapped assets in the model. Find the mapped asset you want to delete and select . The action will delete all the mappings between the model and the asset. # Edit 3D models and revisions Source: https://docs.cognite.com/cdf/3d/guides/3dmodels_edit In Cognite Data Fusion (CDF), you can delete 3D models, and publish, unpublish, and delete revisions. ## View existing 3D models Navigate to **Data fusion** > **Configure** > **3D**. Select a model to see more details. ## Delete a model Navigate to **Data fusion** > **Configure** > **3D**. Select the model you want to delete. In the resource selector, select `...` > **Delete model** and **Delete**. ## Publish, unpublish or delete a revision Navigate to **Data fusion** > **Configure** > **3D**. Select the model you want to view the revisions for. In the resource selector, select the revision you want to publish, unpublish or delete. Select `...` > **Publish** / **Unpublish** or **Delete revision**. # About 3D models Source: https://docs.cognite.com/cdf/3d/guides/3dmodels_intro Upload and optimize large 3D models in Cognite Data Fusion (CDF) to better visualize and contextualize data. CDF automatically analyzes and optimizes the 3D models to enable devices with different form factors to render the models at high frame rates and with great detail. 3D models provide visual and geometrical context to assets. For example, CDF can connect a pump asset with a 3D model of the plant floor where it's placed. Seeing asset data rendered in 3D helps you find the sensor data of your interest faster. Cognite's [REVEAL 3D Web Viewer SDK](https://cognitedata.github.io/reveal-docs/docs) lets you integrate 3D model visualizations in any web-based application. Users can view and interact with the 3D models in a web browser on desktop and mobile devices and in Cognite applications. 3D data is organized in **models** and **revisions**: * **Models** are placeholders for a set of revisions. * **Revisions** contain the 3D data. For example, you can have a model named *Compressor* and upload different revisions of that model. When you create a revision, you need to attach a 3D file. For each new version of the 3D model, you upload a new revision under the same model. Revisions can be **published** or **unpublished** so that applications can decide whether to list the revision. You can publish multiple revisions at the same time, for example, for different versions of the model (high detail vs. low detail). When you upload a new revision, CDF processes the 3D data to optimize it for the renderer. Depending on the complexity of the 3D file, this can take some time. A revision can have the status **Queued**, **Processing**, **Done**, or **Failed**, which you can track during processing. You can upload and process 3D models in the following ways: * [Upload in the CDF user interface](/cdf/3d/guides/3dmodels_upload) * [Upload with File Extractor](/cdf/3d/guides/3d_file_extractor_guide) * [Upload programmatically using the API and SDKs](/dev/3d/api_asset_centric/upload_cad) You can also [edit](/cdf/3d/guides/3dmodels_edit) 3D models in the CDF user interface without writing any code. ## 3D models and project types The way you work with 3D models depends on the type of CDF project you are working on. A project is categorized as either **data modeling** or **asset-centric (legacy)**. In **data modeling** projects, 3D models are stored in Cognite's core data model and are connected to core data model assets. You can only contextualize **core data model assets** to 3D models in these projects. **Asset-centric (legacy)** projects support **hybrid contextualization**, which means you can contextualize both **asset-centric** and **core data model** assets to your 3D models. ## 3D hierarchy and linking to assets 3D data is built up by a hierarchical structure, similar to how CDF organizes its asset hierarchy. Each node in the 3D hierarchy is automatically assigned a unique `nodeId` value. When a user clicks an object in the 3D viewer, the application can get a callback containing the `nodeId` of the clicked object. CDF provides endpoints to extract the full 3D node hierarchy and to create mapping between 3D nodes and nodes in the asset hierarchy. You can use the `nodeId` to connect the 3D data to asset information such as metadata and time series. ## Hardware and system requirements Hardware and system requirements depend on your use case: viewing models in a web browser requires different specifications than processing large CAD or point cloud files. ### Recommended requirements These requirements apply to users viewing and interacting with 3D models through the Reveal 3D viewer in a web browser. **Software requirements:** * **Web browser**: A browser with WebGL2 support, for example, Google Chrome or Mozilla Firefox * **Operating system**: Windows, macOS, Linux iOS and Android devices are also supported for viewing 3D models. **Hardware requirements:** * **Graphics**: Laptops with integrated graphics can be used to view 3D models, for example, Intel Iris Xe Graphics or AMD Radeon Graphics. For the best experience with very large or complex scenes, a dedicated GPU is recommended (NVIDIA GeForce GTX 900 series or newer, or AMD Radeon HD 7000 series or newer, with at least 2 GB VRAM). * **Processor (CPU)**: Intel Core i5 or i7, AMD Ryzen 5/7 series, or similar * **Memory (RAM)**: 16 GB minimum, 32 GB recommended * **Graphics drivers**: Keep your graphics drivers up to date for optimal performance Performance depends on many factors, including hardware, the size of the models you view, and your internet speed. ### Requirements by project size Your hardware needs scale with the complexity and data size of the 3D models in your project: | Project size | Number of nodes (for CAD models) | Number of points (for point clouds) | Example 3D projects | Recommended specifications | | - | - | - | - | - | | **Small** | \<= 500,000 | \<= 100 million | Single equipment pieces, small buildings, individual production units | [Recommended requirements](/cdf/3d/guides/3dmodels_intro#recommended-requirements) | | **Medium** | \<= 3 million | \<= 2 billion | Medium-sized facilities, multi-building sites, partial plant models | 32 GB RAM, GPU with 2 GB+ VRAM, high-core processor recommended for better performance | | **Large** | > 3 million | \<= 10 billion | Entire production facilities, large industrial plants, full offshore platforms, complete infrastructure sites | 32 GB+ RAM, higher-end GPU (4 GB+ VRAM), modern multi-core CPU | ### Limitations **File and processing limits:** * Maximum 3D CAD file size: 100 GiB * Maximum number of nodes: 16.7 million * Maximum size of each node's metadata: 512 KiB (as represented in the list 3D nodes 'properties' JSON object) **Texture visualization:** Textured surfaces can only be visualized for simple textured CAD models with the following constraints: * Only one Albedo / Diffuse texture per node is supported * Textures can be PNG and JPG files * Textures may be scaled down for better performance * UV coordinates outside the \[0, 1] range aren't supported and can result in artifacts CDF doesn't support complex photogrammetry models with heavy textures. **File and processing limits:** * Maximum point cloud file size: 1 TiB * Maximum number of points: 20 billion per model ## Supported 3D file formats | Extension(s) | Type | Version | Description | | - | - | - | - | | DWG (Beta) | CAD | Up to 2024 | [AutoCAD](https://www.autodesk.com/products/autocad/overview) models. | | RVM | CAD | Binary: 1 - 3 | [Aveva E3D and PDMS](https://www.aveva.com/en/products/e3d-design) models. Attributes are supported by providing ATT or TXT files in a zipped folder with the RVM files. See [RVM standard colors for CDF](/cdf/3d/rvm-colors) for color-specific information. RVM ASCII is not supported. | | NWD | CAD | 2012 - 2026 | [Navisworks](https://www.autodesk.com/products/navisworks/overview) models. | | SLDASM, SLDPRT | CAD | 97 - 2025 | [SOLIDWORKS](https://www.solidworks.com) parts and assemblies. Assemblies (.sldasm) and parts (.sldprt) can be combined in a ZIP archive. To upload part files (.sldprt), you must add an assembly file (.sldasm). You can upload assembly files without any part files. | | X\_T, X\_B | CAD | Up to 37.1 | [Parasolid](https://www.plm.automation.siemens.com/global/en/products/plm-components/parasolid.html). Parasolid files can be exported from a range of 3D design tools. | | STP, STEP | CAD | AP 203 E1/E2, AP 214, AP 242 E1/E2/E3 | [STEP](https://en.wikipedia.org/wiki/ISO_10303-21). STEP files can be exported from a range of 3D design tools. | | IFC | BIM/CAD | IFC2x2, IFC2x3 and IFC4 up to 4.3 | [Industry Foundation Classes (IFC)](https://www.buildingsmart.org/standards/bsi-standards/industry-foundation-classes) is an exchange format frequently used in BIM. IFC files can be exported from a range of 3D design tools | | FBX | CAD/General 3D | ASCII: 7100-7500, Binary: All | [Autodesk Filmbox](https://en.wikipedia.org/wiki/FBX) is a generic 3D format. Several 3D tools support this format, in particular Autodesk products. | | OBJ | CAD/General 3D | Any | [Wavefront OBJ](https://en.wikipedia.org/wiki/Wavefront_.obj_file) is a simple text-based 3D format suitable for small models. It can be combined with materials in a MTL file in a ZIP archive. | | PTX | Point cloud | | Text based point cloud scan. Note that binary formats are recommended. | | PTS | Point cloud | | Text based point cloud scan. Note that binary formats are recommended. | | E57 | Point cloud | | [ASTM E57](https://www.astm.org/e2807-11r19e01.html) is an exchange format for 3D image data. See [E57 files with 360 degree images](/cdf/3d/guides/360images_upload#e57-files-with-360-images) for more information. | | LAS | Point cloud | | [ASPRS LAS](https://en.wikipedia.org/wiki/LAS_file_format) is a commonly used format for storing laser scans and point cloud models. | | LAZ | Point cloud | | [LASzip](https://downloads.rapidlasso.de/doc/laszip.pdf) is a lossless compression of LAS files. Because of the compression capabilities, LAZ is a recommended format. | | ZIP | Zip files with CAD/Point cloud files | | Combine multiple model files into a single ZIP archive to combine the models during processing. The supported ZIP archive format is PKZIP. Note that point cloud models cannot be combined with CAD models during processing, but that this is supported in certain Cognite applications and by the [Reveal SDK](https://cognitedata.github.io/reveal-docs). | ## Measurement units in CAD In CAD models, measurement units define the design's size, scale, and accuracy. Measurement units are the standard units (for example, millimeters, inches, and feet) that specify the objects' dimensions within the design. Units ensure the design is correctly scaled and interpreted when shared with others or converted into real-world objects through manufacturing processes. If the source CAD file in CDF has measurement units, it's converted to meters. After conversion, all internal geometry data will be in meters. The Reveal 3D viewer can then load the data correctly since it needs data in metric units like millimeters (mm), centimeters (cm), and meters (m). For CAD metadata, any numeric values aren't converted. They're passed on in a string format. # Upload 3D models in the browser Source: https://docs.cognite.com/cdf/3d/guides/3dmodels_upload Learn to upload 3D models and revisions to Cognite Data Fusion (CDF) using the web interface. CDF supports 3D CAD files from common systems and standard formats such as RVM, OBJ, FBX, and other formats. Point cloud files are also supported in various formats. For more information, see [Supported 3D file formats](/cdf/3d/guides/3dmodels_intro#supported-3d-file-formats). ## Upload a 3D model Navigate to **Data fusion** > **Configure** > **3D**. Select Upload model and follow the wizard to create the model and upload the initial revision of the model. ## Upload a revision to an existing 3D model Navigate to **Data fusion** > **Configure** > **3D**. In the **Models** tab, select the model you want to add a new revision to. In the resource selector, select `...` > **Upload new revision** and follow the wizard to upload a new revision of the model. CDF shows what type of 3D data, CAD or point cloud, is used for each model based on the type of last revision. Use revisions in a 3D model for the same type of 3D data and the physical site. The revisions reflect different points in time in a CAD model or a point cloud model. Use only one type of 3D data within each model ID. ## Further reading To upload multiple 3D models programmatically using our API and SDKs, for instance for nightly updates, see: * [Upload 3D models programmatically](/dev/3d/api_asset_centric/upload_cad) * [Upload point clouds with Python SDK](/dev/3d/api_asset_centric/upload_pointcloud) * [Upload 360° images programmatically](/dev/3d/api_asset_centric/upload_360) The [3D models resource type](/api-reference/concepts/20230101/3d-models) has more information. # Contextualize point clouds Source: https://docs.cognite.com/cdf/3d/guides/pointcloud_contextualize Contextualize point clouds manually or using automated segmentation and object detection in CDF. You can contextualize point clouds in two ways: by manually creating each object you want to contextualize or by leveraging the point cloud's segmentation. You must use both services to achieve a fully contextualized point cloud. ## Before you start Make sure you have already uploaded the point cloud you want to contextualize. For more information on uploading point clouds, see [Upload 3D models](/cdf/3d/guides/3dmodels_upload) or [Upload point clouds with Python SDK](/dev/3d/api_asset_centric/upload_pointcloud). ## Get started Before contextualizing, you should filter and index the point cloud to generate tiles that you can use for object detection and accelerate the contextualization process. It's an automated process triggered by selecting the filter icon in your model's action column. 1. Select the point cloud model and select `...` > **Filtering and indexing**. 2. In the wizard, select the check box to generate tiles. 3. Select **OK** to start the process. Once the filtering and indexing are processed, trigger the object detection. The **Run classification** icon will now be enabled. 1. Select `...` > **Run classification**. 2. Enable both **Classification** and **Pipe detection**. 3. Select **Run**. Classification will perform semantic segmentation of the point cloud, while pipe detection will detect segments in the point clouds that resemble pipes. Once an object is detected, you will see an orange tag on the right, *suggested annotations* indicating how many objects were detected. ## Contextualize Once the filtering and detecting objects are ready, select the point cloud file, and select **Open** to start the contextualization process. This opens the point cloud, and you can contextualize within the 3D environment window. ### Tools in contextualization In the top navigation bar, there is a drop-down menu for all the suggestions. These are suggestions for classifications and pipes. After contextualizing, you will see a drop-down for all the contextualized objects. Use slicing to get a better overview of the model you are contextualizing. For example, you can slice your viewpoint below ceiling height. * You can turn on and off the highlights of the different suggestions of object types from the drop-down. To focus on contextualizing one object type at a time, select **Toggle annotation visibility** in the toolbar on the left to turn off the highlight for all other suggestions. * Select the crop box icon to create crop boxes and slice planes. You can create: * A vertical slice along the X and Y direction. * A vertical slice by selecting two points. * A horizontal slice by selecting a point. * A crop box by selecting four points. * Turn on and off X, Y, and Z axis directions using **Show or hide axis** in the toolbar. * Use the **Settings** icon to adjust the coloring of the points in the model. * You can also color the model by classification. Use the filter icon from the left toolbar. Enable **View classification** to view all or a selection of the classifications. * Select **Fit to view** to reorient yourself and reposition the camera view outside the model. * When you start the contextualization, you will see that the cursor icon is activated and blue. Use **Select Annotation** to select any detected object and annotations. * Select **+** (**Add annotation**) to add an annotation in the point cloud. * After adding an annotation, in the resource selector, use the **Filters** to refine the results. Use to sort and select columns. You can sort by name, object type, location and more. See [Search](/cdf/explore/search) for more information. * To zoom an annotation, select **Zoom to annotation**. ## Contextualize automatically detected objects To link an object to another asset: Navigate to **Suggestions** in the top bar and select an object. In the resource selector on the top right, select **Relink**. Use the search bar in the resource selector. Select the asset, verify it is correct, and select **+ Add**. You have contextualized the object. The annotation count and contextualization count show how many objects you annotated. ## Contextualize undetected objects Point clouds are sets of data points in a three-dimensional coordinate system. Each point in the cloud represents a specific position in space and can contain additional information like color or intensity. This means a point is not related to nearby points, and some objects might not be recognized. To contextualize a group of points as an object, you draw a shape around the points and contextualize that shape. To contextualize an undetected object: 1. Use the icon to add an annotation in the point cloud. 2. Select **+** (**Add annotation**) from the toolbar on the left. In the model, select where you want to contextualize the object. A blue cube will appear in the 3D view. To adjust the cube, use either: * **Translate (T)** will move the cube in the x, y, and z directions. * **Rotate (R)** will rotate the cube. * **Scale (G)** will scale the size of the box. After creating the shape, you can use the resource selector on the right to map an asset to the selected object. Use the search bar to find an existing asset. Select the asset, verify it is correct, and select **+ Add**. Select **Create annotation** in the toolbar at the bottom of the screen. ## Update or delete annotation In point clouds, you have a 1:1 relationship between assets and objects. This means you can only link one asset to a detected object. You can relink, update, or delete an annotation. ### Relink the object to a different asset Use the cursor tool and select the annotated object. Select **Relink** in the top right of the resource selector. Search for the asset you want to map to the selected object. Select **+ Add**. You have updated the annotation. ### Delete annotation To delete the annotation using **Annotations**: Open the **Annotations** dropdown. Search for the asset you want to delete. Hover over each annotation to see the highlighted objects. Select to delete the annotation. You have deleted the annotation. # Set thresholds Source: https://docs.cognite.com/cdf/3d/guides/rule_based_coloring Create rule-based color overlays in 3D models to visualize threshold conditions for assets and time series data. The overlays highlight the objects in the 3D model and change the colors of nodes based on selected thresholds for contextualized data on assets and time series. For example, the 3D nodes are *green* when the metadata, `pipe_pressure` of the linked pipe pressure assets, indicates low values, *yellow* indicates medium values, or *red* when the thresholds are breached. You can create nested thresholds with AND or OR operations. You can also create nested operations and groups of expressions for a specific type of output. You can set thresholds for metadata fields from assets and time series contextualized in 3D. ## Before you start Make sure you create and set up a data model the first time you create a rule. ## Set a threshold 1. Navigate Data fusion > Configure > 3D. 2. Switch to Set thresholds and select **+ Create rule**. 3. Name the threshold. You have now created an empty threshold and can continue setting the rules. Create threshold dialog A threshold can consist of a rule, a group of rules, or nested rules. When you name a threshold, you are naming a threshold that contains rules. When you set a threshold, it consists of triggers that activate when the rule is true. Define the rules for your thresholds. 1. Select **+ Add rule**. 2. If the rule is valid, select the color the overlay should give the 3D node from the **Fill** dropdown. 3. Select the **trigger type**: *Asset metadata string*, *Metadata numeric*, *Time series*, or *Data model*. The data type for a data model supports only `string`, `boolean`, `numeric`, and `date time`. 4. Add the metadata or time series field. When you start typing, you can view the first five items retrieved from CDF based on the metadata prefix. For *data model* trigger type, select the **Data model**, **Category**, and **Property**. 5. Select **Operator**. 6. Define the **Criteria**. 7. Select **+ Add line** to add another line to create nested operations or **Add group** to add another group to build a rule for scenarios with diverse conditions. 8. Select **Save** to publish the rule. The rule is now accessible in search. To view the new rule, navigate to **Search** and select the **Color overlay** menu/settings. Rule based coloring in 3D viewer ### Nested operations To create a nested operation, select **+ Add line**. Nested operations for threshold rules are useful in complex scenarios when: * Multiple conditions must be met simultaneously. * Thresholds depend on variable relationships. * Criteria are hierarchical. * You need to combine logical operators. * Scalability is required. ### Grouped rules By grouping threshold rules, you can easily manage diverse conditions and ensure clarity in situations with different scenarios. To group the thresholds, select **Add group**. It is best if you group the threshold rules for the below scenarios where: * Different sets of conditions govern distinct outcomes or actions. * Thresholds vary based on specific categories or contexts. * Rules need organization for clarity and management. * Complex decision-making requires segmented criteria. * Scalability and modifiability are essential for rule management and updates. You can delete a line, group, or rule. Use the icon across the respective parameter to delete a line, group or rule. # Configure a scene Source: https://docs.cognite.com/cdf/3d/guides/scene Create and configure 3D scenes to view all related 3D data for a physical location in a unified view. You can create a scene by adding all the 3D data available for a location into one space and aligning the different models to position them correctly in the physical space. Viewing the different 3D content in one scene will give the user a richer context and understanding of the physical conditions of their location. You can also add a skybox and a ground plane to the scene. This will make it easier to read the depth and perceive the contrast of the scene. ## Before you start Before creating a scene, make sure you have: * Created and set up the scene data model (required the first time you create a scene for your project). * Uploaded and processed all the 3D data you want to add to your scene. For more information, see [Upload 3D models](/cdf/3d/guides/3dmodels_upload). To use 3D in [Search](/cdf/explore/search#search-in-3d-models), you need to configure a scene and link it to a location. A location is the global [location filter](/cdf/locations/) across Cognite Data Fusion (CDF). ## Create a scene Go to [CDF](https://fusion.cognite.com) > Data fusion > Configure > 3D. Select + Create scene and follow the wizard to create a scene. ## Configure a scene Select the newly created scene to open its overview. Scene overview interface Select **+ Add 3D data** to open the dialog. You can add CAD, point cloud, or 360° images to your scene. Use the search box to find an existing 3D resource, then select **Add**. Added resources appear in the tree view on the left-hand side. Selecting an item from the tree view opens its settings in the right-hand side panel. Select the added model from the tree view and adjust its position, rotation, and scale in the side panel. From the same panel, you can also [contextualize 3D objects](/cdf/3d/guides/scene#unified-3d-contextualization). You can view the scene from different angles using the options in the top bar: * Use the Orbit mode to rotate around the objects to view all the sides. * Use the Fly mode to view the interior and exterior of any closed model. * Double-click anywhere in the scene to move instantly. Custom skybox, ground plane, thumbnail, and minimap uploads are stored as **files** in CDF. How access control applies depends on your [project type](/cdf/3d/guides/3dmodels_intro#3d-models-and-project-types): * **Data modeling** — Uses the scene's space automatically. You can't select a data set in the scene editor. * **Hybrid** — One **data set** per session applies to all custom scene image uploads, not per image. With no data set selected, files are accessible across all data sets. In **hybrid projects**, set the data set before you upload custom images: 1. In the scene side panel, expand Details and select Edit. 2. In Advanced settings, select a data set, or leave No data set selected. 3. Select Save. The data set selection is **not saved** when you publish the scene. It applies only for the current editing session. Set the data set again if you close the scene or start a new session. To create data sets or configure scoped access, see [Create data sets and add data](/cdf/data_governance/guides/datasets/create_data_sets). In the tree view, select Environment. You can configure textures and skyboxes for the selected 3D resource in the side panel. To use custom images for textures, select Add custom and upload your images. Go to the default scene view, for example, by clicking the scene name in the top bar, and expand the Camera accordion in the side panel. Position the camera at the desired view and select **Set camera position**. The camera position is the view a user gets when they enter a scene. If the **Use default camera position for thumbnail** option is turned on, a scene thumbnail is automatically set. In the Camera accordion, you can set the scene thumbnail in three ways: * Select the **Use default camera position for thumbnail** option. * Position the camera to your desired view and select **Set thumbnail**. * Upload a custom image by selecting **Upload custom thumbnail**. Expand the Minimap accordion in the side panel. The minimap is a customizable top-down view that helps you navigate in large scenes. It updates in real time based on the camera position. 1. Select Upload minimap and upload a file that represents a top-down view of your scene, for example, a sitemap or aerial image. 2. Once uploaded, configure the **position**, **rotation**, and **scale** of the minimap to align it with the scene. Double-click a point on the map to jump to that location in the scene. After configuring the minimap, **Publish** the scene to the project. You have published the scene. ## Edit a scene You can modify an existing scene after it's been published, such as updating the scene name, description, geoposition, or linking it to a location. Expand the Details accordion in the side panel of the scene overview. Select Edit and make the necessary changes. Select Save to publish the changes. ## Toolbar tips * **Cropping tool** : slice through the model to view cross-sections or create a crop box to define a custom 3D boundary. * Measurements : measure point coordinates, point-to-point distances, areas (vertical, horizontal, and polygons), volumes, and circle diameters for pipes and tanks. * Settings : set the rendering quality, configure point cloud settings, and change the units. **Default scene settings** You can save your preferred model visibility, rendering quality, point cloud settings, and camera position as defaults. These settings persist across sessions, so you don't need to manually adjust scene configurations each time. * Ghost mode: gray out areas that are out of focus to help you concentrate on specific parts of the model. * Help : view instructions for navigating in a 3D model. ## Contextualize 3D objects You can contextualize 3D objects (CAD, point cloud, or 360° images) within a scene using the unified 3D contextualization editor. To start contextualizing: In the scene tree view, select the **CAD** model, **point cloud**, or **360° image** collection you want to contextualize. In the right-hand panel, select Contextualize. 3D scene overview with contextualization options For detailed steps on contextualizing each type of 3D model, see: * [CAD](/cdf/3d/guides/3dmodels_contextualize) * [Point cloud](/cdf/3d/guides/pointcloud_contextualize) * [360° image](/cdf/3d/guides/360images_contextualize) # About 3D models in CDF Source: https://docs.cognite.com/cdf/3d/index Learn how to upload and contextualize 3D models and create scenes in Cognite Data Fusion. ## Get started Learn the fundamentals of 3D models in CDF. Learn about 3D models, revisions, and supported file formats in CDF. ## Upload Upload CAD models, point clouds, and 360° images to CDF. Upload CAD models and revisions in your browser. Automate 3D model uploads using the file extractor for batch processing. Upload 360° images from E57 files or custom cube map formats. ## Configure and manage Create scenes, set thresholds, and manage your 3D models. Publish, unpublish, or delete 3D models and revisions. Combine multiple 3D models, point clouds, and 360° images into unified scenes. Create rule-based color overlays based on asset metadata and time series data. Display and interact with 3D CAD models from CDF directly in Power BI reports. ## Contextualize Link assets to 3D objects for improved discoverability and navigation. Map assets to 3D objects in CAD models for improved discoverability. Use automated object detection or manual annotations to link point cloud objects to assets. Link 360° images to assets to provide visual context at specific locations. ## Reference Technical reference documentation for 3D models. View color index tables for Aveva RVM files imported into CDF. # CDF color table for Aveva RVM files Source: https://docs.cognite.com/cdf/3d/rvm-colors Reference tables for standard and custom color indices used when importing RVM files to CDF. When you import RVM files, CDF uses the same standard colors as Autodesk Navisworks. If the RVM file doesn't contain an embedded color table, a standard color index is used, see below. The color index references standard color names, where the RGB values are defined in a separate standard RGB table. ## Indexed colors | Index | Color | | - | - | | 1 | Black | | 2 | Red | | 3 | Orange | | 4 | Yellow | | 5 | Green | | 6 | Cyan | | 7 | Blue | | 8 | Magenta | | 9 | Brown | | 10 | White | | 11 | Salmon | | 12 | LightGrey | | 13 | Grey | | 14 | Plum | | 15 | WhiteSmoke | | 16 | Maroon | | 17 | SpringGreen | | 18 | Wheat | | 19 | Gold | | 20 | RoyalBlue | | 21 | LightGold | | 22 | DeepPink | | 23 | ForestGreen | | 24 | BrightOrange | | 25 | Ivory | | 26 | Chocolate | | 27 | SteelBlue | | 28 | White | | 29 | Midnight | | 30 | NavyBlue | | 31 | Pink | | 32 | CoralRed | | 33 | Black | | 34 | Red | | 35 | Orange | | 36 | Yellow | | 37 | Green | | 38 | Cyan | | 39 | Blue | | 40 | Magenta | | 41 | Brown | | 42 | White | | 43 | Salmon | | 44 | LightGrey | | 45 | Grey | | 46 | Plum | | 47 | WhiteSmoke | | 48 | Maroon | | 49 | SpringGreen | | 50 | Wheat | | 51 | Gold | | 52 | RoyalBlue | | 53 | LightGold | | 54 | DeepPink | | 55 | ForestGreen | | 56 | BrightOrange | | 57 | Ivory | | 58 | Chocolate | | 59 | SteelBlue | | 60 | White | | 61 | Midnight | | 62 | NavyBlue | | 63 | Pink | | 64 | CoralRed | | 206 | Black | | 207 | White | | 208 | WhiteSmoke | | 209 | Ivory | | 210 | Grey | | 211 | LightGrey | | 212 | DarkGrey | | 213 | DarkSlate | | 214 | Red | | 215 | BrightRed | | 216 | CoralRed | | 217 | Tomato | | 218 | Plum | | 219 | DeepPink | | 220 | Pink | | 221 | Salmon | | 222 | Orange | | 223 | BrightOrange | | 224 | OrangeRed | | 225 | Maroon | | 226 | Yellow | | 227 | Gold | | 228 | LightYellow | | 229 | LightGold | | 230 | YellowGreen | | 231 | SpringGreen | | 232 | Green | | 233 | ForestGreen | | 234 | DarkGreen | | 235 | Cyan | | 236 | Turquoise | | 237 | Aquamarine | | 238 | Blue | | 239 | RoyalBlue | | 240 | NavyBlue | | 241 | PowderBlue | | 242 | Midnight | | 243 | SteelBlue | | 244 | Indigo | | 245 | Mauve | | 246 | Violet | | 247 | Magenta | | 248 | Beige | | 249 | Wheat | | 250 | Tan | | 251 | SandyBrown | | 252 | Brown | | 253 | Khaki | | 254 | Chocolate | | 255 | DarkBrown | ## RGB color values This table defines the named colors. Color components are in the order Red, Green, and Blue and have a range of 0–100. A value of 100 means 100 percent, which maps to the byte value 255. A percentage value can be converted to the RGB byte value using the formula `byte_value = 255 * percentage_value / 100`. | Color | R | G | B | | - | - | - | - | | Black | 0 | 0 | 0 | | White | 100 | 100 | 100 | | WhiteSmoke | 96 | 96 | 96 | | Ivory | 93 | 93 | 88 | | Grey | 66 | 66 | 66 | | LightGrey | 75 | 75 | 75 | | DarkGrey | 32 | 55 | 55 | | DarkSlate | 18 | 31 | 31 | | Red | 80 | 0 | 0 | | BrightRed | 100 | 0 | 0 | | CoralRed | 80 | 36 | 27 | | Tomato | 100 | 39 | 28 | | Plum | 55 | 40 | 55 | | DeepPink | 93 | 7 | 54 | | Pink | 80 | 57 | 62 | | Salmon | 98 | 50 | 44 | | Orange | 93 | 60 | 0 | | BrightOrange | 100 | 65 | 0 | | OrangeRed | 100 | 50 | 0 | | Maroon | 56 | 14 | 42 | | Yellow | 80 | 80 | 0 | | Gold | 93 | 79 | 20 | | LightYellow | 93 | 93 | 82 | | LightGold | 93 | 91 | 67 | | YellowGreen | 60 | 80 | 20 | | SpringGreen | 0 | 100 | 50 | | Green | 0 | 80 | 0 | | ForestGreen | 14 | 56 | 14 | | DarkGreen | 18 | 31 | 18 | | Cyan | 0 | 93 | 93 | | Turquoise | 0 | 75 | 80 | | Aquamarine | 46 | 93 | 78 | | Blue | 0 | 0 | 80 | | RoyalBlue | 28 | 46 | 100 | | NavyBlue | 0 | 0 | 50 | | PowderBlue | 69 | 88 | 90 | | Midnight | 18 | 18 | 31 | | SteelBlue | 28 | 51 | 71 | | Indigo | 20 | 0 | 40 | | Mauve | 40 | 0 | 60 | | Violet | 93 | 51 | 93 | | Magenta | 87 | 0 | 87 | | Beige | 96 | 96 | 86 | | Wheat | 96 | 87 | 70 | | Tan | 86 | 58 | 44 | | SandyBrown | 96 | 65 | 37 | | Brown | 80 | 17 | 17 | | Khaki | 62 | 62 | 37 | | Chocolate | 93 | 46 | 13 | | DarkBrown | 55 | 27 | 8 | | BlueGrey | 41 | 49 | 58 | # Model with custom color table If you're using Aveva PDMS and your model doesn't follow the color standards, export the model's custom color table along with the RVM file. The RVM file will then include the custom color table. # About language models Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_choosing_lm Select the right language model for your Atlas AI agent based on task complexity, response speed, and cloud availability. Each Atlas AI agent uses **one language model for all its tasks**. The model you choose affects how well the agent reasons through multi-step questions, how quickly it responds, and what it costs to run. Selecting the right model at the start makes it easier to meet your quality and performance targets without having to rebuild the agent. ## Match model to task complexity The main factor in choosing a model is how much reasoning the agent needs to do. An agent that queries the CDF knowledge graph, retrieves time series data, and analyzes the results in a single response needs a model that can reason well across multiple steps. Agents with simpler workflows, like document Q\&A or single-tool lookups, can often perform well with a smaller, faster model. | Task type | Model recommendation | | - | - | | Multi-step queries, multiple tools in sequence | Higher-capability model (for example, GPT-5 series, Claude 4.5 Sonnet, Gemini 2.5 Pro) | | Single-tool lookups, document Q\&A, or summarization | Mid-tier or smaller model (for example, GPT-5 mini, Claude 4.5 Haiku, Gemini 2.5 Flash) | | High-volume agents where response speed matters | Smaller, faster model; verify quality meets your requirements with evaluation test cases | Choose a model that can handle the agent's most demanding tasks. ## Model availability by cloud The models available to your agent depend on the cloud provider and region of your CDF project. Atlas AI supports models from Azure OpenAI, Google Cloud, and AWS. See the [Language model library](/cdf/atlas_ai/references/atlas_ai_agent_language_models) for the full list of available models, their release dates, and their retirement dates. ## Test models with evaluations Start development with a higher-capability model to establish a quality baseline. Once the agent is working well, test whether a smaller model maintains acceptable quality for your use cases. Use your [evaluation test cases](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) to compare responses from different models before making a change. Run your evaluation test cases after every model change, including automatic upgrades at retirement. A model change can affect responses in unexpected ways. ## Model lifecycle Cognite continuously updates the available models. Each stable model has a published retirement date. If you do not migrate your agent to a newer model before the retirement date, Atlas AI automatically upgrades the agent to the latest stable model. To avoid unexpected behavior after an automatic upgrade, monitor the retirement dates in the [Language model library](/cdf/atlas_ai/references/atlas_ai_agent_language_models) and plan migrations before the deadline. After migrating, run your evaluation test cases to confirm the agent's responses meet your expectations. ## Related information * [Language model library](/cdf/atlas_ai/references/atlas_ai_agent_language_models) * [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [About AI agent evaluations](/cdf/atlas_ai/concepts/atlas_ai_agent_evaluation_overview) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) # About debugging and tracing agents Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_debugging Use reasoning traces and tool call logs in Atlas AI to identify why an agent produced an unexpected response and improve its configuration. The reasoning trace and tool call logs show what an Atlas AI agent did for any given response: which tools it called, in what order, what each tool returned, and how the agent used those results. Both are available in Atlas AI for every agent response. Reviewing traces regularly is an important part of running agents in production. An agent that calls multiple tools in sequence can produce a response that looks correct but relies on incomplete or wrong data at one of those steps. The final output alone does not tell you whether the reasoning was sound. ## Reading the reasoning trace Every agent response includes a reasoning field that shows the agent's step-by-step decision-making process. Expand the reasoning field in Atlas AI to see how the agent interpreted the user's question, which tools it selected, and how it combined the results. Use the reasoning trace to answer these questions. * Did the agent load the correct skill for this question? * Did the agent call the tools you expected, in the order the skill specifies? * Did the agent filter on the correct fields and values? * Did the agent use the right operation (for example, `aggregate_instances` rather than `list_instances`) when the question asked for a count or summary? ## Reading tool call inputs and outputs Each tool call in the reasoning trace includes the input the agent sent and the output the tool returned. Select a tool call to expand it and review both. Tool call inputs show you how the agent translated the user's question into an action. Check these points. * Are the filters correct? For example, if the user asked for high-priority work orders, confirm that the agent filtered on `priority = 1` rather than on a text value like `"HIGH"`. * Did the agent query the correct view? * Are the fields in the response the ones you need? Tool call outputs show you what CDF returned. If the output is empty or missing expected data, the issue may be in the data model rather than the agent's configuration. ## Common patterns in agent traces These patterns appear in agent traces when a configuration needs adjustment. ### Agent calls the wrong tool When tool sequencing instructions are absent or unclear in a skill, the agent selects a tool based on the question alone rather than following a documented workflow. Adding explicit sequencing to the relevant skill, specifying which tool to use first and why, gives the agent clearer guidance for questions that involve multiple tools. ### Agent repeats the same tool call When a tool returns no results, some agents retry the same call rather than reporting the absence. A skill instruction that tells the agent to surface empty results rather than retry prevents this loop. ### Agent counts records manually instead of aggregating When a skill or instruction does not specify how to handle count questions, the agent may retrieve all instances and count them manually rather than using `aggregate_instances`. A vocabulary mapping that defines "how many" as a signal to aggregate server-side routes these questions to the correct operation. ### Agent filters on a property that does not exist When property names in a skill or instruction do not match the actual schema because the schema changed or was assumed, the agent constructs queries that return nothing or produce an error. Using `get_view_schema` to verify property names before querying an unfamiliar view, or including explicit property names in the skill's data model guidance, prevents this. ### Agent makes several calls that could be one When query guidance in a skill is too granular, the agent follows each instruction step literally rather than consolidating. Simplifying the guidance to describe the goal and the relevant view, rather than each individual operation, typically resolves this. ## Tracing and evaluation Tracing and [evaluation](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) serve different purposes and can work well together. Tracing shows you what the agent did for a specific response. Use it when a single response is wrong or unexpected and you need to understand why. Evaluation runs a set of test cases against the agent and compares responses to expected outputs. Use it to check that a configuration change improved the responses you intended to improve and did not break other question types. After you identify an issue with tracing and update the agent's skills or instructions, run your evaluation test cases to confirm the fix before publishing. ## Response citations Every Atlas AI agent response includes citations that link the answer back to the data sources the agent used. The product includes citations that appear automatically. You do not need to configure them in instructions or skills. Where tracing shows the full reasoning chain for debugging, citations surface the key sources directly in the response so you can check the underlying records without inspecting the trace. ## Related information * [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [About AI agent evaluations](/cdf/atlas_ai/concepts/atlas_ai_agent_evaluation_overview) * [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) * [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) # About AI agent evaluations Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_evaluation_overview Learn about AI agent evaluations and understand how to measure performance and verify behavior. Atlas AI agent evaluations test how agents respond to specific prompts, like questions about equipment status or maintenance history, to help you identify where to improve agent performance. Use the Cognite CLI and `eval/eval.yaml` for suites you keep in Git and run from the terminal or CI. See [Agent evaluation cases](/dev/sdks/cognite-cli/agents-eval). You can also run evaluations in Cognite Data Fusion (CDF). See [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating). These paths use different workflows and artifacts; don't treat them as interchangeable. ## How evaluations work Evaluations use test cases to verify agent behavior. You create test cases by defining prompts and expected responses. For example, you can create a test case with the prompt, "What is the status of pump P-101?" and an expected response like "Pump P-101 is operational and running at 85% capacity with no active alerts." The expected response specifies information like operational status, performance metrics, and alert status to define what a successful response includes. This diagram shows the workflow for evaluating Atlas AI agents: ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: {"theme":"base","themeVariables":{"primaryColor":"#E4E6E8","primaryTextColor":"#191B1D","primaryBorderColor":"#5E666D","lineColor":"#5E666D","secondaryColor":"#DFE5FC","tertiaryColor":"#FFFFFF","fontFamily":"Inter, system-ui, sans-serif","fontSize":"14px"}}}%% flowchart TD subgraph you["You"] A[Create test cases] B[Run evaluation] H[View results] end subgraph atlasAI["Atlas AI"] C[Send prompts] E[Compare responses] F[Return results] end subgraph agent["Agent"] D[Generate responses] end A --> B --> C --> D --> E --> F --> H classDef youNode fill:#AD4506,stroke:#5E666D,stroke-width:2px,color:#FFFFFF classDef atlasNode fill:#2F55EA,stroke:#5E666D,stroke-width:2px,color:#FFFFFF classDef agentNode fill:#14735E,stroke:#5E666D,stroke-width:2px,color:#FFFFFF class A,B,H youNode class C,E,F atlasNode class D agentNode linkStyle default stroke:#5E666D,stroke-width:2px ``` Test cases define the scenarios your agent should handle and the expected responses for each. When you run evaluations, you can view results to see how your agents perform when Atlas AI compares their responses to your expected responses. These results help you identify patterns, like which types of prompts your agents struggle to understand, so you can improve performance. ## Why evaluations matter When you build agents to solve business problems and automate workflows, evaluations help you verify that they provide the results that you expect. To verify that agents provide the expected results, you can run evaluations at different stages of your agent workflows to monitor performance: * **Check for regression** - Verify that agents still work after you update prompts, add tools, or change configuration. * **Test before deployment** - Verify agents work as expected before you publish them. * **Record expected behavior** - Use test cases to document how agents should respond. * **Track performance over time** - Track how changes affect agent responses to identify where to improve. ## Building effective test cases The value of an evaluation depends on the quality of its test cases. Build your test case library before you start tuning the agent. This gives you a baseline to measure against so you can verify that a change improved the responses you intended to improve without affecting other question types. Follow these practices to build test cases that produce meaningful results. ### Start with the questions users ask most Identify the 10 to 20 prompts that represent your agent's core use cases. Write specific expected responses for each one, including the data points, format, and level of detail you would consider a successful answer. Specific expectations make it easier to identify where the agent's response does not meet the expected output. ### Cover different question types Include simple lookups ("What is the status of Pump P-101?"), multi-step queries ("What maintenance was done on Compressor C-205 in the last 6 months and are there any related open work orders?"), and edge cases such as equipment that does not exist or questions outside the agent's defined scope. ### Re-run after every configuration change When you update skills, modify instructions, change tool configurations, or upgrade the language model, re-run your test cases before publishing. This confirms that changes work as intended and do not affect other question types. ### Use results to guide iteration When an agent's response diverges from the expected output, use the gap to identify what to fix. Unexpected responses often point to missing vocabulary mappings, ambiguous instructions, or skill trigger conditions that need to be more specific. ## Related information * [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Debugging and tracing agents](/cdf/atlas_ai/concepts/atlas_ai_agent_debugging) * [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) # About prompting Atlas AI agents Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_prompting Learn how to write effective prompts to get accurate and relevant responses from Atlas AI agents for your industrial data questions. Atlas AI agents answer questions about your industrial data by querying the CDF knowledge graph. The quality of your prompts directly affects the accuracy and usefulness of agent responses. Specific prompts that include context and define what you need help the agent retrieve the right data and return a useful answer the first time. ## Be specific and clear Specific prompts produce better results than vague ones. Use action verbs to clarify what you want the agent to do. ``` Tell me about Pump P-101. ``` This prompt does not give the agent enough information to focus the response. The agent cannot determine whether you need operational status, maintenance history, specifications, or location information, which can result in an incomplete or unfocused answer. ``` What is the current operational status of Pump P-101? ``` By requesting operational status directly, this prompt helps the agent retrieve the right data and return the specific information you need. ``` List all open high-priority work orders for Compressor C-205 created in the last 30 days. ``` By specifying status, priority, equipment, and time frame, this prompt helps the agent filter and return only the actionable items you need. ## Provide context Context helps agents focus on relevant data by narrowing the scope of your request. ``` Show me the temperature trends. ``` Without knowing which equipment, sensor, or time period to analyze, this prompt leaves the agent to search through data that may not be relevant. You may receive temperature trends for the wrong equipment or an unhelpful time range. ``` Show me the temperature trend for Reactor R-301, specifically for sensor TI-301A, over the past 7 days. ``` By specifying the equipment, sensor, and time frame, this prompt helps the agent retrieve exactly what you need. ``` Considering the recent maintenance on Heat Exchanger HE-502, are there any anomalous pressure readings in the connected lines from the past 48 hours? ``` The maintenance context in this prompt helps the agent distinguish between expected post-maintenance behavior and readings that require attention. ## Define the desired output Specify the format, structure, and level of detail you need in the response. ``` Give me maintenance data for Valve V-789. ``` This prompt does not tell the agent whether you need maintenance history, upcoming schedules, or cost data. It also does not specify a time frame or output format, which can lead to an unfocused or incomplete response. ``` Summarize the maintenance history for Control Valve CV-789 for the last 12 months, focusing on repair actions taken and parts replaced. Present it as a bulleted list. ``` This prompt specifies the data type, time frame, focus, and format, giving the agent everything it needs to produce a useful response. ``` What were the maximum and minimum vibration levels for Motor M-607 on January 15th, 2024? ``` By requesting specific metrics rather than general information, this prompt tells the agent exactly which values to retrieve. ``` List all work orders for Pump P-101 in a table showing work order ID, status, description, and date. ``` Requesting a table with defined columns makes the response easy to scan and compare across multiple items. ## Use schema awareness When your agent uses the Query tool, it can discover the structure of your data model at runtime. You can ask questions that rely on relationships between assets, equipment, and data without knowing the underlying field names or schema. This table shows examples of questions that use schema awareness effectively. | Question | What the agent does | | - | - | | How many maintenance orders are there per equipment type? | Queries the knowledge graph and groups results by equipment type | | What sensors are connected to Compressor C-205, and which ones have readings outside the expected range? | Traverses the relationship between the asset and its linked time series, then checks current values | | Show me all heat exchangers with overdue inspections | Filters on asset type and inspection status without requiring you to specify field names | The more clearly your organization's data model is documented in CDF, the more accurately the agent can answer these types of questions. ## Ask effective follow-up questions Agents retain context within a conversation, so you can build on a previous response without repeating the original question. If the agent returned a list of work orders, you can follow up with "of those, which ones are high priority?" or "show those results as a table" without re-specifying the equipment or time range. Follow-up questions work well when they narrow, reformat, or extend the previous result. If you need to ask about a different piece of equipment or a completely different workflow, starting a new conversation resets the agent's context. When a response is not what you expected, check the reasoning field to see how the agent interpreted your question and which data it used. If the agent queried the wrong equipment or applied the wrong filter, use that information to refine your next prompt with more specific criteria. ## Related information * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Evaluate agents](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) # About Atlas AI agent sandbox and context management Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox Understand the to-do list tracking, sandbox execution environment, automatic context compression, and tool result offloading capabilities available in newer Atlas AI agent runtime versions. These capabilities are available starting with [agent runtime version 2.0.0](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions) (Private preview). Availability is currently limited to select projects. Do not rely on preview features in production systems. Longer, more complex agent tasks, such as analyzing data across thousands of rows or time series data points, can run into a few problems: the task needs to be broken down and tracked, some tools need to run code or commands in a safe environment, and long conversations can grow past the language model's context limit. Runtime version 2.0.0 addresses these with **to-do list tracking**, a **sandbox execution environment**, and **automatic context compression**, so agents can run longer, more complex tasks. ## To-do list tracking For a multi-step task, the agent breaks the work into a to-do list and tracks progress against each item as it works. This produces more reliable results for tasks that span many steps, such as processing a large dataset in batches. ## Sandbox The sandbox gives an agent an isolated, temporary environment to execute commands and run scripts that use the [Cognite Python Software Development Kit (SDK)](/dev/sdks/python/). From the sandbox, an agent can read data from your CDF project but can't create, update, or delete it. Using the sandbox requires the `sessions:create` [capability](/cdf/access/guides/capabilities#atlas-ai). Each command times out after 9.5 minutes. The sandbox stays available for about 12 hours after the last command, then clears automatically. ## Automatic context compression When a conversation grows long, the agent compresses older turns while retaining the details needed to continue the task. This prevents long-running and multi-step tasks from failing when they approach the language model's context limit. No configuration is required. ## Tool result offloading When a tool call returns a large result, such as a lengthy document or extensive query output, the agent stores it separately and keeps only a short preview in context. If the full result is needed later, the agent retrieves it automatically. No configuration is required. ## Related information * [Agent runtime versions](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions) * [About Atlas AI agent subagents](/cdf/atlas_ai/concepts/atlas_ai_agent_subagents) * [About Atlas AI agent tools](/cdf/atlas_ai/concepts/atlas_ai_agent_tools) * [Capabilities](/cdf/access/guides/capabilities#atlas-ai) # About Atlas AI agent skills Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_skills Learn how Atlas AI agent skills give agents domain-specific knowledge and how to use them to get accurate, consistent results. Skills are structured documents that give an Atlas AI agent domain-specific knowledge for a particular task or dataset. When a user's question is relevant to a skill, the agent loads that skill and uses its guidance to answer. The agent does not load skills that are irrelevant to the current question, which keeps the agent's context focused on what matters. ## What skills contain A skill tells the agent how to approach a specific type of question. A well-written skill includes these elements. * **Trigger description.** A short statement that tells the agent when to load this skill, helping it decide whether the skill is relevant to the user's question. * **Data model guidance.** Instructions on which views to query and which properties are important. The agent can discover the schema at runtime, but explicit guidance reduces errors on frequently-used query patterns. * **Vocabulary mappings.** A mapping between the terms users are likely to use and the field values stored in CDF. Vocabulary mappings are one of the most effective ways to improve agent accuracy because ambiguous terminology is a common source of incorrect queries. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} When the user says "high priority," filter on priority = 1. When the user says "overdue," filter where plannedEndDate is in the past and status is not "COMPLETED". When the user says "rotating equipment," include assets with tag prefixes P- (pumps), C- (compressors), and T- (turbines). ``` * **Tool sequencing instructions.** The order in which the agent should call tools when answering questions in this skill's domain. For example, when a question asks about sensor trends, the agent should query the knowledge graph to identify the relevant time series, retrieve data points, and then analyze them if the user asks about trends or anomalies. * **Response templates.** Instructions on what a complete answer looks like, including which fields to include and how to format the output. * **Examples.** A small set of diverse examples covering distinct question patterns. A few well-chosen examples are more effective than a long list of similar ones. ## How skills relate to agent instructions Agent instructions (the system prompt) define how the agent behaves in every interaction: its tone, language, safety boundaries, and output format defaults. Skills provide domain-specific knowledge that the agent loads on demand. Keeping these separate means the agent's context stays focused on what matters for the current question. When domain knowledge loads on every interaction regardless of relevance, the agent processes far more information than it needs and may miss important instructions. Domain knowledge only enters the context when it is relevant to the current question. Workflow-specific guidance, vocabulary mappings, and tool sequencing belong in skills, not in the agent's general instructions. ## Skills as reusable building blocks A skill is not tied to a specific agent. A skill that teaches an agent how to navigate your asset hierarchy, interpret naming conventions, or query your maintenance data model can be attached to any agent that works with that data. This makes skills a building block for your agent library. A data scouting skill that retrieves sensor readings, alarm history, and recent work orders for a piece of equipment is just as useful in a root cause analysis agent as it is in an equipment health agent. Writing skills as self-contained units, without references to a specific agent's identity or purpose, makes them straightforward to reuse. ## Keeping skills focused A skill that tries to cover too many topics at once loses effectiveness. Each skill should focus on one domain or workflow step. A root cause analysis agent, for example, might use three separate skills: one for gathering relevant data, one for identifying correlations and anomalies in that data, and one for building a causal chain from symptom to root cause. If a skill grows long enough to become difficult to read, consider splitting it into two focused skills with clear, distinct triggers. ## Related information * [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [About Atlas AI agents](/cdf/atlas_ai/concepts) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) # About Atlas AI agent subagents Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_subagents Understand how subagents let an Atlas AI agent delegate tasks to other agents, and when to use them to build orchestrator-style workflows. Subagents (Preview) are available starting with [agent runtime version 1.3.0](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions). Do not rely on preview features in production systems. Subagents let an Atlas AI agent delegate a task to another independently configured agent. The primary agent, sometimes called the orchestrator, decides when a task fits a subagent's purpose, passes relevant context to it, and uses the result to continue the conversation. The subagent runs with its own language model, instructions, skills, and tools rather than inheriting the orchestrator's configuration. ## Why use subagents **Build an orchestrator agent.** Instead of one agent trying to handle every workflow, an orchestrator agent can route a question to an existing specialized agent, or to one you build specifically to support the orchestrator. This keeps each agent focused on a narrow, well-tested task. **Isolate context.** A subagent runs in its own context. The orchestrator's conversation isn't cluttered with the full detail of how a sub-task was completed, and the subagent isn't distracted by instructions, skills, or tools that are irrelevant to its task. **Match the language model to the task.** A subagent can use a different language model than its orchestrator. Route simple, well-defined sub-tasks to a faster or cheaper model to speed up the overall system, and reserve a more advanced model for subagents that handle more complex reasoning. ## Adding a subagent Add subagents from the **Subagents** section in the Agent builder. The agent you want to delegate to must have a description. ## Approval and client-side tools A subagent can use tools that require user approval or run client-side. When one of these tools is called during a delegated task, the approval request surfaces in the orchestrator's conversation, labeled with the subagent's name, so the user can approve or deny it without leaving the top-level conversation. ## Limits * An agent supports up to 20 subagents. * An agent can't delegate to itself. * An agent can't add the same subagent twice. * A subagent can't itself delegate to other agents. Delegation is limited to one level. ## Related information * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [About Atlas AI agent tools](/cdf/atlas_ai/concepts/atlas_ai_agent_tools) * [Agent runtime versions](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions) # About Atlas AI agent tools Source: https://docs.cognite.com/cdf/atlas_ai/concepts/atlas_ai_agent_tools Understand how Atlas AI agent tools extend what agents can do and why routing computation to tools produces more accurate results than relying on language model reasoning. Atlas AI agents use tools to retrieve data, perform calculations, and interact with external systems. A language model can reason about a question and decide what to do, but it needs tools to act on those decisions. When the agent calls a tool, it passes structured inputs, receives a result, and uses that result to build its response. ## Why the right set of tools matters Each tool you enable adds to the decisions the agent must make. More tools means more opportunities for the agent to choose the wrong one. An agent with a focused tool set makes more reliable decisions than one with a long list of overlapping options. Enable only the tools your agent's workflow requires. Use skills to document which tool to use and when, so the agent does not have to infer the correct choice at runtime. Some tools work as pairs. Time series tools and document tools need the Query tool to run first. The agent uses the Query tool to identify the right asset or file in the knowledge graph, then uses the specialized tool to retrieve or analyze its data. When you enable these tools together, add sequencing instructions to your skills so the agent knows which tool to call first. For example, if a user asks whether a bearing temperature is trending upward, the agent first uses the Query tool to find the relevant time series in the knowledge graph. It then retrieves the readings with Query time series data points and calculates the trend with Analyze time series. Without sequencing instructions in the skill, the agent may skip the first step or retrieve readings for the wrong time series. ## Use tools for calculations Language models are strong at reasoning and language understanding. They are not reliable for exact arithmetic, date calculations, or precise comparisons. When a question has a correct, verifiable answer, a tool should compute it, not the language model. This table shows common question types and the tool that produces a reliable result. | Question type | Route to | | - | - | | "How many work orders are open?" | `aggregate_instances` in the Query tool, which computes the count server-side rather than listing all records and counting them in context | | "Is the vibration on Compressor C-205 trending upward?" | Analyze time series, which performs statistical analysis rather than asking the language model to estimate a trend | | "What percentage of inspections are overdue at each site?" | `aggregate_instances` in the Query tool, or Run Python code for multi-step calculations the Query tool does not cover | | "Of those work orders, which ones took longer than planned?" | Examine data semantically, which analyzes results already retrieved in the same conversation without re-querying the knowledge graph | | "Run a domain-specific calculation that requires custom packages" | Call Function, which runs on Cognite's cloud infrastructure with full package support and longer execution times | You can reinforce this routing behavior in skills. When the user asks "how many," "what percentage," or "compare X to Y," a skill can instruct the agent to use the appropriate tool rather than reason about the answer directly. See [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) for examples. ## Tool categories Atlas AI tools fall into three categories. **Query tools** retrieve data from the CDF knowledge graph. The [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) is the foundation for most agents: it works across your entire data model without requiring separate configuration per view. The legacy query knowledge graph tools each target a single view. **Analysis tools** work with data that has already been retrieved or identified. Time series tools retrieve sensor data and perform statistical analysis. Document tools answer questions from files and generate summaries. The Examine data semantically tool analyzes result sets from earlier queries in the same conversation. **Integration tools** let the agent run custom code or call external services. [Run Python code](/cdf/atlas_ai/references/atlas_ai_agent_python_tool) runs a builder-defined script in a WebAssembly sandbox. [Call Function](/cdf/atlas_ai/references/atlas_ai_call_function_tool) calls a deployed Cognite Function on Cognite's cloud infrastructure. For the full list of tools and their configuration options, see the [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools). ## Choosing between Run Python code and Call Function Both tools run custom Python code, but each fits different situations. [Run Python code](/cdf/atlas_ai/references/atlas_ai_agent_python_tool) runs a script you define in a sandboxed environment with access to CDF APIs and standard libraries. Use it for lightweight calculations that stay within CDF and do not need external network access or custom packages. [Call Function](/cdf/atlas_ai/references/atlas_ai_call_function_tool) calls a deployed Cognite Function on Cognite's cloud infrastructure. Use it when you need custom packages, more memory, longer execution times, or access to systems outside CDF. If a calculation fits within the sandbox, Run Python code can be simpler to set up. If it needs anything beyond those limits, use Call Function. ## Related information * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) * [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) * [Run Python code](/cdf/atlas_ai/references/atlas_ai_agent_python_tool) * [Call Function](/cdf/atlas_ai/references/atlas_ai_call_function_tool) * [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) # About Atlas AI agents Source: https://docs.cognite.com/cdf/atlas_ai/concepts/index Learn what makes a good Atlas AI agent use case and how to configure agents that are accurate, trustworthy, and useful in production. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). Atlas AI agents query data stored in CDF's Data Modeling Service (DMS). Your CDF project must use DMS and the Cognite core data model before you can configure an agent that queries the knowledge graph. Atlas AI agents respond to your prompts by retrieving, analyzing, and summarizing data from CDF. They can access assets, equipment records, sensor readings, work orders, and documents. They can also search across all of those sources at once and combine the relevant information into a single response. ## Identify the right use case Not every workflow benefits from an agent. Good use cases involve questions that require judgment, draw on multiple data sources, and rely on accurate, well-documented data. These dimensions help you assess whether a problem is suitable for an agent before you start building. ### Accuracy Language models can give different answers to the same question. Any use case that requires every answer to be exactly right, such as payroll calculations or safety-critical permit approvals, is not suitable for a fully autonomous agent. This does not prevent using agents as part of those workflows, but critical decision points require your review. Workflows where some variation is acceptable, or where you review the output before acting, are more suitable. ### Risk Consider what happens when the agent is wrong. Some tasks have a low tolerance for error but also low consequences if something goes wrong, making them suitable for early experimentation. For tasks where an error could affect safety, regulatory compliance, or production, you should review the agent's output before taking action. ### Volume and availability Agents are most useful for high-volume or always-on tasks. Tasks like reviewing maintenance procedures, interpreting inspection reports, or extracting specific parameters from large volumes of vendor documentation are suitable for an agent. Tasks that require 24/7 availability, such as monitoring operational logs or supporting control room questions, also work well. ### Predictability Predictable, repeatable processes typically work better with traditional automation. Agents are most useful in open-ended workflows where inputs vary and each question requires different steps to answer. ### Documentation and data readiness Agents have no built-in knowledge of your organization. You need to give the agent context through clear documentation that describes the task, the relevant data, and the rules that apply. Use cases that rely on undocumented knowledge are unlikely to work. The data the agent queries must also be accurate and well-structured. Data quality is not optional. Agents can handle specific parts of many real-world processes, even when you still need to monitor the overall workflow. Identifying those parts can be more practical than trying to automate the entire process. ## Define the agent's scope and instructions Each agent should focus on one workflow or question type, such as equipment fault diagnosis, work order triage, or document Q\&A. Keeping the agent's mission narrow makes it easier to test, maintain, and trust. The skills that support that mission can be as detailed as the domain requires. The agent's instructions define how it behaves in every interaction. Good instructions cover four things. Scope and identity defines what the agent does and what it should decline to handle. Universal rules apply to every question regardless of topic, such as stating assumptions when data is ambiguous. Output expectations describe what a complete response looks like. Boundaries establish what the agent must not do without user confirmation. Workflow-specific knowledge belongs in skills, not instructions. The Agent builder walks you through writing and testing instructions before you publish. Step-by-step walkthrough of creating and publishing an agent in the Agent builder. Available runtime versions, feature support by version, and how to upgrade. ## Choose a language model Each agent uses one language model for all its interactions. The model you choose affects how well the agent handles multi-step questions, how quickly it responds, and how much it costs to run. Start with a higher-capability model to establish a quality baseline, then test whether a smaller model meets your requirements once the agent is working well. How to match the right model to your agent's complexity and performance needs. All available models, release dates, and retirement timelines by cloud region. ## Equip the agent with the right tools Each tool you enable adds to the decisions the agent must make. Enable only the tools the agent's workflow requires. The Query tool is the foundation for most agents. Other tools extend it for time series data, documents, and custom computation. Tools that work with time series or documents rely on the Query tool to find the right asset or file first, so configure these combinations as pairs and document the expected order in your skills. How to select tools, use them for computation, and combine them correctly. Browse all available tools by category with configuration options for each. Configuration reference for the Query tool, including operations and example queries. How to write and configure custom Python scripts for agent computations. How to connect an agent to a deployed Cognite Function. ## Build focused skills Skills give the agent domain knowledge for specific workflows. Each skill contains vocabulary mappings, data model guidance, and tool sequencing instructions, and loads only when the user's question is relevant. This keeps the agent's context focused on what matters for the current question. For example, a root cause analysis agent might use three separate skills: one for gathering relevant sensor data, alarms, and work order history; one for identifying patterns and anomalies in that data; and one for building the causal chain from symptom to root cause. Each skill can be tested and improved on its own, and all three could be reused in an equipment health agent without modification. What skills contain and how they give agents domain knowledge on demand. How to write, structure, and test skills using the skill builder or manually. ## Delegate with subagents When one agent cannot cover every workflow without losing focus, use subagents (Preview). An orchestrator agent routes a task to another independently configured agent that runs with its own language model, instructions, skills, and tools. Requires runtime version 1.3.0 or later. When to use subagents, how to add them in the Agent builder, and their current limits. ## Manage context and sandbox Longer, more complex tasks, such as analyzing data across thousands of rows or time series data points, can run into a language model's context limit or need a safe place to execute commands. Runtime version 2.0.0 (Private preview) adds to-do list tracking, a sandbox execution environment, automatic context compression, and tool result offloading, so agents can handle these tasks without extra configuration. How to-do list tracking, the sandbox, automatic context compression, and tool result offloading work. ## Test early and iterate Define test cases before you start tuning the agent. A set of 10 to 20 representative questions with specific expected responses gives you a baseline to measure against. Run your test cases after every meaningful change to skills, instructions, tools, or the language model to catch problems early. How test cases, evaluations, and results help you measure and improve agent quality. Step-by-step guide to creating test cases and running an evaluation. ## Trace and debug agent behavior Testing shows you what went wrong. Tracing shows you why. Every agent response includes a reasoning field that shows which tools the agent called, in what order, and how it used the results. Inspecting this chain is important for agents that call multiple tools in sequence, where a problem in one step can produce a final response that looks correct but is based on incomplete or wrong data. How to use reasoning traces and tool call logs to diagnose and fix agent issues. ## Query the agent Once published, you can query the agent through the Atlas AI interface or call it through the API and SDK. Specific, context-rich questions produce better results than vague ones. Techniques for writing specific, context-rich queries that produce better results. How to call agents programmatically and handle tool confirmations. # Using Atlas AI agents via API and SDK Source: https://docs.cognite.com/cdf/atlas_ai/guides/atlas_ai_agent_api_sdk Use the Cognite Python SDK or REST API to chat with Atlas AI agents, continue conversations with a cursor, and approve or deny tool confirmations for integration tools. You can call Atlas AI agents from the [Cognite Python SDK](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/agents.html) or the [Agents REST API](/api-reference/concepts/20230101-beta/ai-agents) to embed agents in your applications, automate workflows, and build custom user interfaces. The Agents REST API is in *beta* release status. Python SDK coverage may lag the API and is subject to change. ## Prerequisites * A deployed Atlas AI agent. See [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building). * The `agents:run` and `agents:read` [capabilities](/cdf/access/guides/capabilities). * Python SDK version 7.80.3 or later: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} pip install "cognite-sdk>=7.80.3" ``` ## Chatting with an agent Send a message to an agent and read the response: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.agents import Message client = CogniteClient() response = client.agents.chat( agent_external_id="my_agent", messages=Message("What can you tell me about compressor C-101?"), ) print(response.text) ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -X POST "https://{cluster}.cognitedata.com/api/v1/projects/{project}/ai/agents/chat" \ -H "Authorization: Bearer {token}" \ -H "Content-Type: application/json" \ -d '{ "agentExternalId": "my_agent", "messages": [ { "role": "user", "content": { "type": "text", "text": "What can you tell me about compressor C-101?" } } ] }' ``` ## Continuing a conversation Use the `cursor` from the previous response to continue the same conversation. You do not need to resend previous messages; the cursor maintains conversation state: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} follow_up = client.agents.chat( agent_external_id="my_agent", messages=Message("What are the related time series?"), cursor=response.cursor, ) print(follow_up.text) ``` ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "agentExternalId": "my_agent", "cursor": "", "messages": [ { "role": "user", "content": { "type": "text", "text": "What are the related time series?" } } ] } ``` ## Handling tool confirmation When an agent uses an integration tool ([Call Function](/cdf/atlas_ai/references/atlas_ai_call_function_tool), [Run Python code](/cdf/atlas_ai/references/atlas_ai_agent_python_tool), or Call REST API), the response may include a **tool confirmation request** before the tool runs. This prevents the agent from performing destructive or unintended actions without your approval. The confirmation appears as a `toolConfirmation` action in the response. Your application must respond with `ALLOW` to proceed or `DENY` to cancel. In the Python SDK, `response.action_calls` is `None` when the response contains no actions, so check for it before you iterate over it. ### The confirmation flow Your message causes the agent to run an integration tool. The response contains an action with `type: "toolConfirmation"` instead of the final result. It includes the tool name, arguments, and a human-readable description of what the tool will do. Your application reviews the tool call details and sends back `ALLOW` to execute it or `DENY` to cancel it. The agent executes (or skips) the tool and returns the final result. ### Example ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.agents import Message, ToolConfirmationCall, ToolConfirmationResult client = CogniteClient() # Step 1: Send the initial message response = client.agents.chat( agent_external_id="my_agent", messages=Message("Run the data quality check function"), ) # Step 2: Check for tool confirmation requests if response.action_calls: confirmations = [] for action in response.action_calls: if isinstance(action, ToolConfirmationCall): print(f"Agent wants to run: {action.tool_name}") print(f"Tool type: {action.tool_type}") print(f"Arguments: {action.tool_arguments}") print(f"Description: {action.tool_description}") confirmations.append( ToolConfirmationResult( action_id=action.action_id, status="ALLOW", # or "DENY" to cancel ) ) # Step 3: Send ALLOW or DENY if confirmations: response = client.agents.chat( agent_external_id="my_agent", messages=confirmations, cursor=response.cursor, ) # Step 4: Print the final result print(response.text) ``` **Step 1: Initial request** ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "agentExternalId": "my_agent", "messages": [ { "role": "user", "content": { "type": "text", "text": "Run the data quality check function" } } ] } ``` **Step 2: Response with confirmation request** ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "agentExternalId": "my_agent", "response": { "type": "result", "cursor": "", "messages": [ { "role": "agent", "content": { "type": "text", "text": "" }, "actions": [ { "actionId": "id_22c74316-1014-47e8-ae2b-ed0767c6c471", "type": "toolConfirmation", "toolConfirmation": { "content": { "type": "text", "text": "Please confirm the action." }, "toolName": "Data quality check", "toolType": "callFunction", "toolArguments": {}, "toolDescription": "This tool calls a Cognite Function." } } ] } ] } } ``` **Step 3: Send ALLOW or DENY** ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "agentExternalId": "my_agent", "cursor": "", "messages": [ { "role": "action", "type": "toolConfirmation", "actionId": "id_22c74316-1014-47e8-ae2b-ed0767c6c471", "status": "ALLOW" } ] } ``` Include the `cursor` from the confirmation response in your ALLOW or DENY request so the agent continues the same conversation. ## Related information * [Call Function tool reference](/cdf/atlas_ai/references/atlas_ai_call_function_tool) * [Run Python code tool reference](/cdf/atlas_ai/references/atlas_ai_agent_python_tool) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Agents API reference](/api-reference/concepts/20230101-beta/ai-agents) * [Python SDK: Agents](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/agents.html) # Building agents Source: https://docs.cognite.com/cdf/atlas_ai/guides/atlas_ai_agent_building Step-by-step guide to create Atlas AI agents with the Cognite CLI or the Agent builder in CDF. Build Atlas AI agents that solve business problems and automate workflows. Use the **Cognite CLI** for a file-based workflow you can version in Git, review in pull requests, and reuse across projects. You can also build in the low-code **Agent builder** in Cognite Data Fusion (CDF). Both paths produce agents you can publish to the **Agent library** for your CDF project. ## Before you start Complete these steps before building your agent. 1. **Define your agent's scope.** Each agent should handle one specific workflow or question type, such as equipment fault diagnosis, work order triage, or document Q\&A. 2. **Identify an evaluation dataset.** Gather 10 to 20 representative questions your users will ask, along with the answers you would consider correct, to use as test cases after building. 3. **Choose a language model.** Select the [language model](/cdf/atlas_ai/references/atlas_ai_agent_language_models) that fits your agent's task complexity. See [About language models](/cdf/atlas_ai/concepts/atlas_ai_agent_choosing_lm) for guidance. ## Choose how to build Build with the **Cognite CLI** or the **Agent builder** in Cognite Data Fusion (CDF). Use `@cognite/cli` to scaffold an agent project (`.agent.yaml`, evaluation cases, and coding-assistant guidance), push drafts to CDF, run evaluations, and publish. Follow the guide linked below. * [Building Atlas AI agents with the Cognite CLI](/dev/sdks/cognite-cli/agents) * [Agent definition (`.agent.yaml`)](/dev/sdks/cognite-cli/agent-yaml) * [Command reference: agents](/dev/sdks/cognite-cli/reference#agents) Requires `@cognite/cli` **1.10.0** or later. Use the low-code Agent builder in CDF for interactive configuration. In CDF, navigate to **Atlas AI** > **Agent builder**. Select **+ Create agent** or select a template to use as a starting point. Enter a **Name** that clearly describes the agent's purpose. Write a **Description** that explains what problems the agent solves or tasks it automates. Add **Sample prompts** that show users how to interact with the agent. Make these specific and representative of real use cases. Select the [**Language model**](/cdf/atlas_ai/references/atlas_ai_agent_language_models) you identified earlier. Write **Instructions** that define the agent's behavior for every interaction. Effective instructions cover four elements. Define what the agent is for and what requests it should decline. You are a maintenance assistant for rotating equipment. You help engineers diagnose faults, review work order history, and retrieve sensor data. You do not provide recommendations on staffing or procurement. Write rules that apply to every interaction, regardless of what the user asks. If the data is incomplete or ambiguous, say so explicitly rather than making assumptions. State your assumptions when more than one interpretation is possible. Describe what a complete, well-formed response should include. When listing work orders, always include the work order ID, description, priority, status, and planned start date. State what the agent must not do without explicit confirmation from the user. Do not create or modify work orders. Do not recommend specific maintenance actions. Present findings and let the engineer decide. Keep instructions focused on universal behavior. Workflow-specific guidance, vocabulary mappings, and tool sequencing belong in skills, which the agent loads on demand when relevant. [Skills](/cdf/atlas_ai/concepts/atlas_ai_agent_skills) give the agent domain-specific knowledge for specific workflows. Use a separate skill for each distinct workflow or dataset so the agent loads only what is relevant to the current question. 1. In the Agent builder, select **Skills**, then **+ Add skill**. 2. Use the **skill builder** to generate a skill from an existing document such as an SOP, engineering specification, or naming convention guide. Review and adjust the generated draft before saving. You can also write skills manually. 3. Attach the skill to your agent and test with questions that should trigger it. For guidance on structuring skills, see [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills). Add [**Tools**](/cdf/atlas_ai/references/atlas_ai_agent_tools) to let the agent access CDF data, perform calculations, or interact with other applications. * **Start with the Query tool.** The [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) works across your entire data model without requiring separate configuration per view. * **Enable only the tools your agent needs.** Each additional tool increases the chance the agent selects the wrong one. * **Document tool combinations in skills.** When a question requires more than one tool, add sequencing instructions to the relevant skill rather than relying on the agent to infer the order. See [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) for examples. For the **data retrieval** tools, specify which **access scope** to use. | Option | What it does | | - | - | | Inherit scope from the user's location | Uses the [location](/cdf/locations) configured in CDF to determine which data to find. | | Inherit scope based on user's access rights | Finds data without filtering on [spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space). | | Follow access scope defined manually | Limits the query to the [spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space) you select. | Agent actions inherit CDF access controls. The agent can only access data the user has permission to view. Optionally add [subagents](/cdf/atlas_ai/concepts/atlas_ai_agent_subagents) (Preview) so this agent can delegate tasks to other specialized agents. Add them from the **Subagents** section in the Agent builder. Requires runtime version 1.3.0 or later. Test the agent in Atlas AI. Refine the language model, instructions, tools, and skills as needed. Test with questions that cover a range of scenarios. * Typical questions your users will ask * Questions that involve multiple tools or data types * Edge cases, such as equipment that does not exist or questions outside the agent's scope * Follow-up questions that build on a previous response When a response is unexpected, expand the reasoning field and tool call logs to see what the agent did. See [Debugging and tracing agents](/cdf/atlas_ai/concepts/atlas_ai_agent_debugging) for guidance on reading traces. Run the test cases you gathered before you started. Use [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) in CDF, or [Agent evaluation cases](/dev/sdks/cognite-cli/agents-eval) if you want a file-based suite in Git. Select **Publish** to make the agent available in the **Agent library** to everyone in your CDF project. Monitor the agent's performance and update its skills, instructions, and tools based on user feedback and usage patterns. Re-run your evaluation test cases after any configuration change to confirm that existing behavior is not affected. ## Related information * [Building Atlas AI agents with the Cognite CLI](/dev/sdks/cognite-cli/agents) * [About Atlas AI agents](/cdf/atlas_ai/concepts) * [About Atlas AI agent skills](/cdf/atlas_ai/concepts/atlas_ai_agent_skills) * [Configuring skills for agents](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills) * [About Atlas AI agent subagents](/cdf/atlas_ai/concepts/atlas_ai_agent_subagents) * [About language models](/cdf/atlas_ai/concepts/atlas_ai_agent_choosing_lm) * [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [Debugging and tracing agents](/cdf/atlas_ai/concepts/atlas_ai_agent_debugging) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) # Configuring skills for agents Source: https://docs.cognite.com/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills Write and structure Atlas AI agent skills to give agents domain-specific knowledge for accurate, consistent responses. Skills give Atlas AI agents domain-specific knowledge for a particular task or dataset. ## Before you start You must have an agent configured in the [Agent builder](/cdf/atlas_ai/guides/atlas_ai_agent_building) before you can attach skills to it. ## Creating a skill The Agent builder includes a skill builder that generates a structured skill from a description or source document. You can also write skills manually in markdown. In CDF, navigate to **Atlas AI** > **Agent builder** and open your agent. Select **Skills**, then select **+ Add skill**. To generate a skill automatically, paste in or upload source material relevant to the skill's domain. Suitable sources include SOPs, engineering specification documents, naming convention guides, and equipment manuals. The skill builder extracts the structure, vocabulary, and rules from the source material and produces a draft skill. This can be faster than writing a skill manually and works well for skills that cover naming conventions, tag formats, and asset hierarchies. If you provide an existing document, review the generated skill for accuracy before saving. The skill builder produces a structured draft, but domain-specific rules may require adjustment. Review each section of the generated skill and complete any sections that need adjustment. See [Structuring a skill](#structuring-a-skill) for guidance on each section. Save the skill and attach it to your agent. Test the agent with questions that should trigger the skill to verify that it loads correctly and improves the agent's responses. Use [agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) to confirm behavior against expected responses before publishing. ## Structuring a skill A well-structured skill contains these sections. Not every skill requires all of them, but each section you include should be specific and accurate. The trigger description tells the agent when to load this skill. Write it as a direct statement that identifies the relevant question type, equipment category, or data domain. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Use this skill when the user asks about maintenance orders, work orders, or planned maintenance for rotating equipment, including pumps, compressors, and turbines. ``` A clear trigger prevents the agent from loading the skill for unrelated questions. Tell the agent which views to query and which properties are relevant for this skill's domain. The Query tool can discover the schema at runtime, but explicit guidance reduces errors on frequently-used query patterns and helps the agent choose correctly when a data model has multiple date or status fields with similar names. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Work orders are stored in the MaintenanceOrder view in the Cognite process industries data model. Key properties: - priority: integer, where 1 = high, 2 = medium, 3 = low - status: string, for example "OPEN", "IN_PROGRESS", "COMPLETED" - plannedStartDate: the scheduled start of the work - actualStartDate: the date work actually began - asset: a direct relation to the Equipment view Use plannedStartDate when the user asks when work is scheduled. Use actualStartDate when the user asks when work began. ``` When a view has multiple fields with similar names or unclear descriptions, add a note explaining which field to use for each question type. For guidance on making your data model clearer for agents, see [Optimizing data models for AI](/cdf/dm/dm_guides/dm_best_practices_ai_search). Add mappings that translate the terms users are likely to use into the field values stored in CDF. This is one of the most effective ways to improve accuracy, because unclear terminology is a common source of errors in industrial data queries. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Vocabulary: - "high priority" = priority = 1 - "medium priority" = priority = 2 - "overdue" = plannedEndDate is in the past AND status != "COMPLETED" - "open" = status = "OPEN" OR status = "IN_PROGRESS" - "rotating equipment" includes pumps (tag prefix P-), compressors (C-), and turbines (T-) ``` Document the order in which the agent should call tools when answering questions in this skill's domain. These tabs show sequencing instructions for three common tool combinations. When a user asks about sensor readings, trends, or anomalies on a piece of equipment, the agent should follow this sequence. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} 1. Use the Query tool to identify the relevant time series for the equipment in the knowledge graph. Look for time series linked to the asset with a description or tag that matches the sensor type the user is asking about (for example, vibration, temperature, pressure). 2. Use the Query time series data points tool to retrieve the values for the requested period. If the user does not specify a time period, default to the last 24 hours. 3. If the user asks about trends, anomalies, or comparisons to a previous period, use the Analyze time series tool to perform the calculation. Do not estimate trends from raw values. ``` When a user asks a question that refers to a procedure, manual, specification, or inspection report, the agent should follow this sequence. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} 1. Use the Query tool to find the relevant file in the knowledge graph. Search for files linked to the equipment the user is asking about, and filter by document type or title if the user specifies one. 2. Confirm the file name and its relationship to the asset before proceeding. 3. Use Answer document questions to retrieve the answer from the file's contents. Quote the relevant excerpt in your response so the user can verify the source. ``` When a user asks a follow-up question about data that was already retrieved in the same conversation, the agent should use the Examine data semantically tool rather than querying the knowledge graph again. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} If the user asks a follow-up question about previously retrieved results (for example, "of those work orders, which ones were delayed?"), use Examine data semantically to analyze the existing result set. Do not issue a new knowledge graph query for follow-up questions that refer to data already in the conversation. This avoids duplicate queries and keeps the response focused on the results the user has already seen. ``` Tell the agent what a complete answer looks like for this skill's domain. Include which fields to show and any formatting preferences. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} When listing work orders, always include the following fields: - Work order ID - Description - Priority (display as High, Medium, or Low rather than as a number) - Status - Planned start date - Asset name Present the results as a table when listing multiple work orders. If there are more than 10 results, summarize the total count and highlight the highest-priority items. ``` Include examples that show the agent how to handle the most common question patterns for this skill. A few examples covering distinct scenarios are more effective than a long list of similar ones. Cover different question types: a simple lookup, an aggregation, a multi-tool question, and an ambiguous request. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Example 1: Simple lookup User: Show me the last 5 work orders for Pump P-101. Action: Query the MaintenanceOrder view, filter on asset = P-101, sort by plannedStartDate descending, limit to 5 results. Example 2: Aggregation User: How many open work orders are there for each equipment type? Action: Use the aggregate_instances operation to count open work orders grouped by asset type. Do not retrieve all work orders and count them manually. Example 3: Multi-tool User: Are there any active alarms on Compressor C-205 and is there a related open work order? Action: Query the knowledge graph for active alarms linked to C-205, then query for open work orders linked to the same asset. Summarize both sets of results together. Example 4: Ambiguous request User: Fix this pump. Action: Explain that you cannot take action on physical equipment. Retrieve the current status, open work orders, and recent alarms for the pump, and suggest that the user review the findings and raise a work order if needed. ``` ## Limitations | Limit | Details | | - | - | | File format | Single `.md` file only. Zip archives and associated files are not supported. | | File size | 30 KB maximum. | | Name format | Alphanumeric characters and hyphens (`-`) only. No other symbols are allowed. | | Reserved name prefix | Names must not start with `cdf-`. This prefix is reserved for Cognite packaged skills. | ## Related information * [About Atlas AI agent skills](/cdf/atlas_ai/concepts/atlas_ai_agent_skills) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) * [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) * [Optimizing data models for AI](/cdf/dm/dm_guides/dm_best_practices_ai_search) # Running agent evaluations Source: https://docs.cognite.com/cdf/atlas_ai/guides/atlas_ai_agent_evaluating Step-by-step guide to test Atlas AI agents with the Cognite CLI or Evaluate agents in Cognite Data Fusion (CDF). Configure evaluations to verify that your agents provide the results that you expect. Create test cases with prompts and expected responses, run evaluations to generate agent responses, then review results to identify where your agents can improve. Use the **Cognite CLI** for evaluation cases you keep in Git and run from the terminal or CI. You can also run evaluations with **Evaluate agents** in Cognite Data Fusion (CDF). These paths use different workflows and artifacts; don't treat them as interchangeable. ## Before you start You must create an agent with the [Cognite CLI](/dev/sdks/cognite-cli/agents) or in the [Agent builder](/cdf/atlas_ai/guides/atlas_ai_agent_building). CDF stores evaluations created with Evaluate agents as data model instances in the `atlas-ai-evals` space. Your CDF group must have the `datamodelinstances:write` capability scoped to that space before you can create an evaluation. See [Atlas AI capabilities](/cdf/access/guides/capabilities#atlas-ai). ## Choose how to evaluate Evaluate with the **Cognite CLI** or **Evaluate agents** in CDF. Define cases in `eval/eval.yaml`, score with `cognite agents eval`, and keep suites in version control. * [Agent evaluation cases](/dev/sdks/cognite-cli/agents-eval): scorers, schema, session context, and tool scorers * [Building Atlas AI agents with the Cognite CLI](/dev/sdks/cognite-cli/agents): scaffold → push → evaluate → publish * [Evaluation cases JSON Schema](/assets/schemas/agent-eval-cases.schema.json): for editors and coding assistants Requires `@cognite/cli` **1.10.0** or later. Use **Evaluate agents** in CDF for interactive test cases in the browser. ### Configure evaluation Configure an evaluation with test cases that define prompts and expected responses. 1. In CDF, navigate to **Atlas AI** > **Evaluate agents**. 2. Select **+ Create new evaluation**. 1. Enter a **Name** for your evaluation. Use descriptive names like "Pump maintenance agent - v1" to help you identify your evaluations. 2. Add a **Description** to explain the purpose of the evaluation. Add test cases to define prompts and the expected responses. 1. In the **Test cases** section, enter a **Prompt**. 2. Enter the **Expected response**. Atlas AI compares your expected response to the agent's response to test performance. 3. Optionally, choose an agent and select **Generate answer** to use the agent to generate the expected response. Be specific about what details, like operational status, metrics, or alert information, you want agents to include in their responses. ### Run and monitor evaluation Run an evaluation to test your agent. Monitor real-time progress as the agent processes test cases. 1. Go to **Evaluation overview**. 2. Select **Run evaluation**. 1. Use the search or filter to find your agent by name, status, or latest version. 2. Select the agent from the list. 3. Select **Run with selected agent** to run the evaluation. If Atlas AI prompts you to allow code tool access for your agent, select **Confirm and run** to continue running the evaluation or **Cancel** to go back to the list of agents. When you run the evaluation, Atlas AI sends your prompts to the agent and records the responses in real time. This can take several minutes depending on how many test cases you have. You can select **Cancel run** to stop the evaluation at any time. Do not close the browser tab while the evaluation is running. Evaluations run in the browser, and closing or navigating away from the tab will stop the evaluation. ### View and analyze results Review evaluation results to identify where agents need improvement and track performance across multiple runs. On the **Evaluation overview** tab, review the list of completed evaluations. Select an evaluation to view the status of your test cases. Select **View details** to compare the agent responses to your expected responses for each test case. Use evaluation insights to improve your agent by refining instructions in the Agent builder, updating tools or data access, or editing test cases. ## Related information * [Agent evaluation cases](/dev/sdks/cognite-cli/agents-eval) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [About AI agent evaluations](/cdf/atlas_ai/concepts/atlas_ai_agent_evaluation_overview) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) # Building Atlas AI agents Source: https://docs.cognite.com/cdf/atlas_ai/index Create and manage industrial AI agents that query your CDF knowledge graph using language models, skills, and tools. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). Choose an article below based on where you are in the agent lifecycle — from planning and building to evaluation and production use. Learn what makes a good agent use case and how agents use language models, skills, and tools. Language models, tools, runtime versions, and project limits. Create and publish agents in the Agent builder. Create and push agent files, run evaluations, and deploy through version control and CI/CD. Create test cases and run evaluations to measure agent quality. Write specific, context-rich queries that produce better agent responses. Query agents programmatically and handle tool confirmations. # Language model library Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_agent_language_models Detailed reference of available language models in Atlas AI, including stable, legacy, and retired models with release and retirement dates. Choose the right model based on your requirements for speed, quality, and context length. The availability of language models depends on your preferred cloud vendor and [the location](/cdf/admin/clusters_regions) of your Cognite Data Fusion (CDF) project. Cognite continuously refreshes language models with newer and more capable models for building agents. As part of this process, Cognite retires older models. These terms describe lifecycle stages and important dates for language models available in Atlas AI: * **Stable model**: A released model version **supported for production use**. Stable models include a retirement date when Cognite Atlas AI will no longer support them. * [**Latest stable model**](#latest-stable-models): The most current version within the model family, recommended for new agents or agents in development. Use this model when migrating from older versions. * **Legacy stable model**: An older model version that the latest stable model has replaced. While Atlas AI still supports legacy models, migrate to the latest model to access new features and improvements. * **Recommended upgrade**: The latest stable model to migrate to for improved performance and additional features. * **Retired model**: A model version that Atlas AI has permanently deactivated after reaching its retirement date. Atlas AI no longer supports retired models. If you don't migrate agents using retired models before the retirement date, they automatically upgrade to the latest stable model. ### Latest stable models | Model ID | Release date | Retirement date | | - | - | - | | azure/gpt-5.6-sol | August 2026 | No earlier than June 24, 2027 | | azure/gpt-5.6-terra | August 2026 | No earlier than June 24, 2027 | | azure/gpt-5.6-luna | August 2026 | No earlier than June 24, 2027 | | azure/gpt-5.5 | May 2026 | No earlier than April 8, 2027 | | azure/gpt-5.4 | March 2026 | No earlier than February 18, 2027 | | azure/gpt-5.4-mini | May 2026 | No earlier than March 3, 2027 | | azure/gpt-5.4-nano | May 2026 | No earlier than March 3, 2027 | | azure/gpt-5.2 | March 2026 | No earlier than April 29, 2027 | | azure/gpt-5.1 | January 2026 | No earlier than October 29, 2026 | | azure/gpt-5 | August 2025 | No earlier than July 15, 2026 | | azure/gpt-5-mini | August 2025 | No earlier than July 15, 2026 | | azure/gpt-5-nano | August 2025 | No earlier than July 15, 2026 | | azure/gpt-4.1 | August 2025 | No earlier than April 1, 2026 | | azure/gpt-4.1-nano | August 2025 | No earlier than April 1, 2026 | | azure/gpt-4.1-mini | August 2025 | No earlier than April 1, 2026 | | gcp/claude-5-sonnet | July 2026 | No earlier than January 9, 2027 | | gcp/claude-4.8-opus | July 2026 | No earlier than May 13, 2027 | | gcp/claude-4.7-opus | May 2026 | No earlier than August 31, 2027 | | gcp/claude-4.6-sonnet | May 2026 | No earlier than June 30, 2027 | | gcp/gemini-3.5-flash | July 2026 | No earlier than May 4, 2027 | | gcp/claude-4.5-sonnet | September 2025 | No earlier than September 14, 2026 | | gcp/claude-4.5-haiku | September 2025 | No earlier than September 16, 2026 | | gcp/gemini-2.5-pro | August 2025 | No earlier than June 1, 2026 | | aws/claude-5-sonnet | July 2026 | No earlier than January 9, 2027 | | aws/claude-4.7-opus | April 2026 | No earlier than August 31, 2027 | | aws/claude-4.6-sonnet | April 2026 | No earlier than June 30, 2027 | | aws/claude-4.5-sonnet | September 2025 | No earlier than September 14, 2026 | | aws/claude-4.5-haiku | September 2025 | No earlier than September 16, 2026 | ### Legacy stable models | Model ID | Release date | Retirement date | Recommended upgrade | | - | - | - | - | | | | | | ### Retired models | Model ID | Release date | Retirement date | Recommended upgrade | | - | - | - | - | | gcp/gemini-1.5-pro | December 2024 | September 15, 2025 | gemini-2.5-pro | | gcp/gemini-1.5-flash | December 2024 | September 15, 2025 | gcp/gemini-3.5-flash | | gcp/gemini-2.0-flash | February 2025 | September 15, 2025 | gcp/gemini-3.5-flash | | gcp/gemini-2.5-flash | August 2025 | June 2, 2026 | gcp/gemini-3.5-flash | | aws/claude-3-haiku | December 2024 | September 15, 2025 | aws/claude-5-sonnet | | aws/claude-4-opus | August 2025 | January 13, 2026 | aws/claude-4.5-sonnet | | aws/claude-4.1-opus | August 2025 | January 13, 2026 | aws/claude-4.5-sonnet | | aws/claude-3.5-sonnet | December 2024 | February 15, 2026 | aws/claude-5-sonnet | | aws/claude-4-sonnet | May 2025 | May 7, 2026 | aws/claude-5-sonnet | | azure/gpt-4o | December 2024 | March 16, 2026 | azure/gpt-4.1 | | azure/gpt-4o-mini | December 2024 | March 16, 2026 | azure/gpt-4.1-mini | | azure/o3 | August 2025 | March 27, 2026 | azure/gpt-5.6-sol | | azure/o4-mini | August 2025 | March 27, 2026 | azure/gpt-5.6-luna | # Run Python code Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_agent_python_tool Reference information for using the Run Python code tool in Atlas AI agents, including supported parameter types, code structure requirements, and best practices The **Run Python code** tool enables Atlas AI agents to run custom Python scripts with runtime arguments. The tool automatically validates your code, generates contracts for the agent from Python type annotations, and provides testing capabilities within the Atlas AI interface. ## Required code structure To ensure your script works with the Atlas AI agent, you must define specific entry points and follow the required function signatures. ### The handle() function Every Python script must include a `handle()` function as the entry point. Name the function `handle`, annotate all parameters with their types, and include a docstring following the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#383-functions-and-methods). ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle(param1: str, param2: int) -> str: """ Brief description of what the function does. Args: param1 (str): Description of the first parameter. param2 (int): Description of the second parameter. Returns: str: Description of what is returned. """ # Your implementation here return f"Processed {param1} with value {param2}" ``` **Requirements:** * Name the function `handle`. * Annotate all parameters with their types. See [Supported parameter types](#supported-parameter-types). * Include a docstring following the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#383-functions-and-methods). * The return type annotation is recommended but not required. * The function can be synchronous (`def handle()`) or asynchronous (`async def handle()`). The AI agent uses only the parameter descriptions from the `Args:` section of the docstring. It does not use the function description or return value description. To provide descriptions for the agent on how to use the tool, use the Instructions field in the agent configuration interface. ### Async handle() function You can define `handle()` as an async function using `async def` to perform asynchronous operations. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import asyncio async def handle(value: int) -> int: """Async handle function.""" await asyncio.sleep(0.1) return value * 2 ``` **When to use async handle:** * **Eventual consistency**: When you create a resource and need to wait before retrieving it to ensure it's available. * **Delayed operations**: When you need to introduce delays between operations. * **Async SDK operations**: When using async methods from libraries that support async/await patterns. **Requirements for async handle:** * Use `async def` instead of `def` for the function definition. * Use `await` for asynchronous operations like `asyncio.sleep()`. * All the same requirements apply as for synchronous `handle()` functions (type annotations, docstrings, etc.). * The same parameter types and return types are supported for both sync and async functions. ### The test() function Include a `test()` function for validation and testing. Use the Test code button in the Atlas AI agent builder to trigger the function. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def test(): """Test function that runs when you click 'Test code' in the UI.""" # You can use assertions assert handle("test", 42) == "Processed test with value 42" # Or return a value to display in the console return handle("example", 100) ``` **Requirements:** * Name the function `test`. * No parameters are allowed. * The function can be synchronous (`def test()`) or asynchronous (`async def test()`). * The function can use assertions or return values for display. * The function runs when you click the Test code button in the UI. ### Async test() function You can define `test()` as an async function to test async operations or await async `handle()` functions: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import asyncio async def handle(value: int) -> int: """Async handle function.""" await asyncio.sleep(0.1) return value * 2 async def test(): """Async test function that tests async handle.""" # Await the async handle function directly result = await handle(21) assert result == 42, f"Expected 42, got {result}" return result ``` **When to use async test:** * When testing async `handle()` functions you can await them directly. * When you need to perform async operations in your test setup or teardown. * When testing scenarios that involve delays or eventual consistency. ## Supported parameter types Parameters can be of primitive types, `DateTime`, `NodeId` and lists of these. Primitive types can have a default value. ### Primitive types | Python type | JSON schema type | Example usage | Runtime value | | - | - | - | - | | `str` | `"string"` | `name: str` | `"John Doe"` | | `int` | `"integer"` | `age: int` | `25` | | `float` | `"number"` | `temperature: float` | `23.5` | | `bool` | `"boolean"` | `is_active: bool` | `true` | ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle( name: str, age: int, temperature: float, is_active: bool ) -> dict: """ Example with primitive types. Args: name (str): User's name. age (int): User's age in years. temperature (float): Temperature in Celsius. is_active (bool): Whether user is active. Returns: dict: Processed user data. """ return { "name": name, "age": age, "temperature": temperature, "is_active": is_active } ``` ### Default values Primitive types support default values: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle( message: str, count: int = 1, multiplier: float = 1.0, enabled: bool = True ) -> str: """ Example with default values. Args: message (str): The message to process. count (int): Number of repetitions (default=1). multiplier (float): Multiplier factor (default=1.0). enabled (bool): Whether processing is enabled (default=True). Returns: str: Processed message. """ if not enabled: return "Processing disabled" return (message * count) * int(multiplier) ``` ### List types We support lists of primitive types: ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle( names: list[str], values: list[int], temperatures: list[float] ) -> dict: """ Example with list parameters. Args: names (list[str]): List of names. values (list[int]): List of integer values. temperatures (list[float]): List of temperature readings. Returns: dict: Summary of the lists. """ return { "name_count": len(names), "total_value": sum(values), "avg_temperature": sum(temperatures) / len(temperatures) if temperatures else 0 } ``` We don't support nested lists (e.g., `list[list[str]]`). ### DateTime type For temporal data use `datetime`: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import datetime def handle( start_time: datetime.datetime, end_time: datetime.datetime, timestamps: list[datetime.datetime] ) -> dict: """ Example with datetime parameters. Args: start_time (datetime.datetime): Start of time range. end_time (datetime.datetime): End of time range. timestamps (list[datetime.datetime]): List of timestamps. Returns: dict: Time range information. """ duration = end_time - start_time return { "duration_seconds": duration.total_seconds(), "timestamp_count": len(timestamps) } ``` ### NodeId type To work with Cognite Data Fusion (CDF) instances from data models, use [`NodeId`](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_modeling.html) to identify the instances: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client.data_classes.data_modeling import NodeId def handle(asset_id: NodeId, related_asset_ids: list[NodeId]) -> dict: """ Example with NodeId parameters. Args: asset_id (NodeId): Primary asset identifier with space and external_id. related_asset_ids (list[NodeId]): List of related asset identifiers. Returns: dict: Asset information. """ # NodeId objects have .space and .external_id properties return { "primary_asset": { "space": asset_id.space, "external_id": asset_id.external_id }, "related_count": len(related_asset_ids) } ``` **Runtime format:** The tool passes `NodeId` parameters to the language model as objects with `space` and `externalId` properties: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "space": "my-space", "externalId": "asset-123" } ``` The `NodeId` class uniquely identifies nodes in Cognite's data modeling framework, combining a `space` (workspace identifier) and `external_id` to form a unique node reference. See the [Cognite Python SDK data modeling documentation](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_modeling.html) for details. ### Limitations and restrictions The following types are **not supported** as arguments to `handle()`: * Custom classes and objects * Nested lists (`list[list[str]]`) * Complex generics beyond `list[T]` * Union types (`str | int`) * Optional types (`Optional[str]`) * Tuples, sets, and other collections * String literal types (e.g., `Literal["A"]`, `Literal["A", "B"]`) The following restrictions apply for default argument values for `handle()`: * Default values are only supported for primitive types (`str`, `int`, `float`, `bool`). * Lists cannot have default values. * `NodeId` and `datetime` parameters cannot have default values. ## Supported return types Your `handle()` function can return any JSON-serializable data. The following return types are supported: ### Primitive return types ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle() -> str: return "Success message" def handle() -> int: return 42 def handle() -> float: return 3.14159 def handle() -> bool: return True ``` ### Dictionary returns ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle() -> dict: return { "status": "completed", "count": 10, "results": ["item1", "item2"] } ``` ### List returns ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle() -> list[str]: return ["result1", "result2", "result3"] def handle() -> list[dict]: return [ {"id": 1, "name": "Item 1"}, {"id": 2, "name": "Item 2"} ] ``` ### None/null returns ```python theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle() -> None: # Perform an action and return None print("Operation completed") return None ``` ### Unsupported return types The following return types are **not supported**: * Raw Cognite SDK objects * Custom class instances * Functions or callable objects * File handles or binary data * Complex objects that can't be serialized to JSON ## Cognite SDK and authentication The Cognite Python SDK comes pre-installed and configured with your session credentials. The tool uses your current credentials and requires no additional configuration. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteAsset def handle() -> dict: """ List assets. Returns: dict: Asset information including ID and metadata. """ # Client is automatically configured - no setup required client = CogniteClient() # List assets assets = client.data_modeling.instances.list(sources=[CogniteAsset.get_source()]) if assets: return assets[0].dump() # Use .dump() for JSON serialization return {"error": "No assets found"} ``` For complete SDK usage and API reference, see the [Cognite Python SDK documentation](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/). ### Cognite SDK return types We don't support direct serialization of Cognite SDK objects. To reduce token usage and make the data readable to the agent, extract relevant values and reshape the data as needed. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteAsset def handle() -> list[dict]: """ Get simplified asset information. Returns: list[dict]: List of assets with only essential fields. """ client = CogniteClient() assets = client.data_modeling.instances.list( sources=[CogniteAsset.get_source()], limit=10 ) # Extract only relevant fields return [ { "external_id": asset.external_id, "name": asset.name, "description": asset.description } for asset in assets ] ``` As a last resort, you can use the `.dump()` method to serialize all information from Cognite SDK objects, but this may include unnecessary data that increases token usage. ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle() -> dict: """Get complete asset data using dump().""" from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteAsset client = CogniteClient() assets = client.data_modeling.instances.list(sources=[CogniteAsset.get_source()]) return assets[0].dump() # ✅ Correct - JSON serializable def handle() -> list[dict]: """Get all assets using dump().""" from cognite.client import CogniteClient from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteAsset client = CogniteClient() assets = client.data_modeling.instances.list(sources=[CogniteAsset.get_source()]) return assets.dump() # ✅ Correct - JSON serializable ``` ## When to use Run Python code vs Call Function Both the **Run Python code** tool and the **Call Function** tool let your agent run custom Python logic. They differ in where the code runs and what capabilities are available. | Capability | Run Python code | Call Function | | - | - | - | | Code location | Written and maintained in the Agent builder | Deployed separately as a Cognite Function | | Execution environment | WebAssembly sandbox (Pyodide, Python 3.11) | Cognite cloud infrastructure | | Custom packages | Not supported | Supported via `requirements.txt` | | External network access | Not supported (CDF APIs only) | Supported | | Execution time | Limited | Up to the configured max polling time | | Memory | Limited by browser memory | More headroom for larger datasets | Use Run Python code when you need a focused calculation or data transformation that works within the WebAssembly sandbox. The code is stored in the agent configuration, making it easy to write and test. This tool works well for domain-specific logic like unit conversions, threshold checks, or formatting results from Query tool output. Use Call Function when the logic requires custom packages, external network calls, or longer execution times. A Cognite Function runs on Cognite's cloud infrastructure, so it can access third-party APIs, use packages from PyPI, and handle larger datasets. This tool requires the function to be deployed separately before the agent can call it. If you are not sure which to use, start with Run Python code. Switch to Call Function if you encounter package or environment limitations. ## Best practices We recommend the following best practices when writing Python code for the **Run Python code** tool. ### Error handling Use descriptive error messages: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle(value: float) -> float: """ Process a value with validation. Args: value (float): Input value to process. Returns: float: Processed value. """ if value < 0: raise ValueError("The value must be non-negative.") # Clear, actionable message if value > 1000: raise ValueError("The value exceeds the maximum allowed limit of 1000.") return value * 2 ``` ### Documentation * Always include docstrings following the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#383-functions-and-methods). * **Parameter descriptions are critical**: Only the `Args:` section descriptions are passed to the AI agent as part of the JSON schema. * Write clear, specific parameter descriptions that help the agent understand what values to provide. * Include examples in the parameter descriptions when helpful. **Example of parameter documentation for the agent:** ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def handle(asset_id: NodeId, threshold: float) -> dict: """ Analyze asset performance against threshold. Args: asset_id (NodeId): The specific asset to analyze. Provide both space and externalId. threshold (float): The performance threshold value in percentage (0-100). Returns: dict: The analysis results with performance metrics. """ # Implementation here pass ``` ### Testing Write comprehensive test functions. You can use both synchronous and asynchronous test functions: ```python wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} def test(): """Comprehensive test coverage.""" # Test normal cases result1 = handle("test", 42) assert result1 == expected_value, "Normal case failed" # Test edge cases result2 = handle("", 0) assert result2 is not None, "Edge case failed" # Test error conditions try: handle("invalid", -1) assert False, "Should have raised ValueError" except ValueError: pass # Expected # Return a sample result for display return handle("sample", 100) ``` ## Troubleshooting These are common errors when using the **Run Python code** tool, and how to fix them. ### "Function 'handle' not found" * Ensure your function is named `handle`. * Check for indentation errors. ### "Missing type annotation for parameter(s)" * Make sure all parameters have type annotations. * Use supported types only. ### "Duplicate argument name(s) found" * Make sure the parameter names are unique. * Check for copy-paste errors. ### "Unsupported type for parameter" * Use only [supported types](#supported-parameter-types). * Check for typos in type annotations. ### Serialization errors * Use the `.dump()` method on Cognite SDK objects. * Make sure the return values are serializable to JSON. ## Code execution environment capabilities and limitations The **Run Python code** tool runs in a controlled WebAssembly environment with specific capabilities and limitations: * **Runtime**: Pyodide (Python 3.11 in WebAssembly). * **Execution context**: Runs in a web worker for isolation. * **Performance**: WebAssembly execution may be slower than native Python. * **Memory**: Limited by browser memory constraints. ## Pre-installed packages The following packages are automatically available: * **Cognite Python SDK** (`cognite-sdk`): Pre-configured with authentication. * **Standard library**: Full Python 3.11 standard library. * **Core scientific packages** (via Pyodide): * `numpy`: Numerical computing * `pandas`: Data analysis and manipulation * `json`: JSON encoding/decoding * `datetime`: Date and time handling * `uuid`: UUID generation We don't support installing custom packages. This includes: * `pip install` commands * Dynamic package installation * Custom or third-party packages not included in Pyodide If you need custom packages or more advanced functionality, consider using [Cognite Functions](/cdf/functions/) instead. Functions run on Cognite's cloud infrastructure and support: * Custom package installation via `requirements.txt` * Longer execution times * More memory and compute resources * Full Python ecosystem access ## System limitations The execution environment has certain constraints that you should be aware of when writing your scripts. * **File system**: No access to local file system or file I/O operations. The scripts have access to a virtual memory-based file system. * **Network access**: Only API calls to Cognite Data Fusion APIs are supported. Network calls to other servers might get blocked. * **Process control**: No subprocess or system command execution. * **Threading**: Limited threading capabilities in WebAssembly environment. * **Graphics**: Matplotlib backend set to 'AGG' (non-interactive). ## Use via API or SDK When you call an agent with this tool enabled through the REST API or Python SDK, the agent will request confirmation before running the Python code. Your application receives a `toolConfirmation` action in the response and must reply with `ALLOW` to proceed or `DENY` to cancel. See [Use Atlas AI agents via API and SDK](/cdf/atlas_ai/guides/atlas_ai_agent_api_sdk) for a complete example. ## How the AI agent uses your code The Atlas AI agent processes your Python code through these steps to understand and execute your tool functions: 1. **JSON schema as tool descriptor**: The generated JSON schema is passed to the AI agent's language model as a tool descriptor. This tells the agent exactly what parameters are available, their types, and what they're for. 2. **Parameter descriptions guide the agent**: The agent uses the parameter descriptions from your docstring's `Args:` section to understand what values to provide. 3. **Type validation**: The schema ensures the language model provides correctly typed arguments that match your function signature. 4. **Runtime execution**: When the agent calls your tool, it validates the arguments against the schema and transforms them (for example, creating NodeId objects from dictionaries) before passing them to your `handle()` function. Your parameter docstrings are critical. They directly influence how the agent understands and uses your tool. # Agent runtime versions Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions Complete reference for Atlas AI agent runtime versions including stable, preview, and retired versions with feature support and retirement dates. Agent runtime versions define the capabilities and behavior of your Atlas AI agents. Each version includes a specific system prompt, agent tools and their prompts, and agent-specific features. When you create an agent, Atlas AI automatically pins it to the latest stable runtime version. The agent uses a specific runtime version to provide stability for production deployments. Cognite regularly releases improvements to Atlas AI, but your agent maintains consistent behavior with the assigned runtime version. You control when to upgrade your agent to a newer version. ## Lifecycle stages These terms describe lifecycle stages for runtime versions: * **Stable version**: A released runtime version that **supports production use**. Each stable version includes a retirement date when support ends. The most current stable version is marked **(Latest)** — use it for new agents, agents in development, or when migrating from older versions. * **Legacy version**: An older stable version that is supported but has been replaced by a newer stable version. Migrate to the latest version to access new features and improvements. * **Preview version**: A pre-release version that provides early access to some features. Do not use these versions in production systems. * **Private preview version**: A preview version currently limited to specific projects selected by Cognite. Availability expands as Cognite validates the version. * **Retired version**: A runtime version that is no longer supported and has reached its retirement date. Atlas AI automatically upgrades agents using a retired version to the latest stable version if you do not migrate them before the retirement date. ## Agent runtime version overview This table lists all available runtime versions with their current status and support timeline. | Version | Status | Release date | Retirement date | | - | - | - | - | | 1.3.0 | Stable (Latest) | June 2, 2026 | No earlier than June 2, 2027 | | 1.2.0 | Stable | May 5, 2026 | No earlier than May 5, 2027 | | 1.1.1 | Legacy | February 24, 2026 | No earlier than February 24, 2027 | | 1.1.0 | Legacy | February 24, 2026 | No earlier than February 24, 2027 | | 1.0.2 | Legacy | November 27, 2025 | No earlier than November 27, 2026 | | 1.0.1 | Legacy | November 27, 2025 | No earlier than November 27, 2026 | | 1.0.0 | Legacy | November 27, 2025 | No earlier than November 27, 2026 | | 0.9.9 | Legacy | November 27, 2025 | No earlier than November 27, 2026 | | 2.0.0 | Private preview | August 17, 2026 | No earlier than October 17, 2026 | | 2.0.0-alpha | Preview | November 27, 2025 | - | ### Stable version features This table lists the features available in each stable version. | Feature | 1.3.0 | 1.2.0 | 1.1.1 | 1.1.0 | | - | - | - | - | - | | Query tool (Public preview) | ✅ | ✅ | ❌ | ❌ | | Subagents (Preview) | ✅ | ❌ | ❌ | ❌ | | [Query memory](#query-memory) | ✅ | ✅ | ❌ | ❌ | | [Examine data semantically](/cdf/atlas_ai/references/atlas_ai_agent_tools#analysis-tools) | ✅ | ✅ | ❌ | ❌ | ### Preview version features This table lists the features available in preview versions. | Feature | 2.0.0 | | - | - | | Subagents | ✅ | | To-do list tracking | ✅ | | Sandbox (Private preview) | ✅ | | Automatic context compression | ✅ | | Tool result offloading | ✅ | Do not use preview versions in production systems. Features and behavior may change without notice. ## Feature overview These features vary between runtime versions. ### Query tool (Public preview) Runtime versions 1.2.0 and 1.3.0 support the Query tool, which lets agents query Data Modeling directly with explicit operations for view discovery and instance reads. It can list available views, inspect view schemas, and run list, search, aggregate, and retrieve calls on instances with configurable data model and instance-space scoping. ### Subagents (Preview) Runtime version 1.3.0 supports subagents, which allow an Atlas AI agent to delegate tasks to other configured agents. See [About Atlas AI agent subagents](/cdf/atlas_ai/concepts/atlas_ai_agent_subagents) for when to use subagents and their current limits. ### To-do list tracking Runtime version 2.0.0 supports to-do list tracking, so an agent breaks a multi-step task into a list and tracks progress against each item. See [About Atlas AI agent sandbox and context management](/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox) for details. ### Sandbox Runtime version 2.0.0 gives agents a sandboxed environment to execute commands and run scripts, with read access to your CDF project. See [About Atlas AI agent sandbox and context management](/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox) for details. ### Automatic context compression Runtime version 2.0.0 automatically compresses long conversations to stay within the language model's context limit. See [About Atlas AI agent sandbox and context management](/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox) for details. ### Tool result offloading Runtime version 2.0.0 automatically stores large tool results outside the conversation and retrieves them again when needed. See [About Atlas AI agent sandbox and context management](/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox) for details. ### Query memory Query memory is a caching capability that allows agents to store results from [Query tools](/cdf/atlas_ai/references/atlas_ai_agent_tools#query-tools) and pass the results as input to other tools in the same conversation. Some tools require Query memory to function, such as the [Examine data semantically](/cdf/atlas_ai/references/atlas_ai_agent_tools#analysis-tools) tool. ### Examine data semantically This feature enables you to add the [Examine data semantically tool](/cdf/atlas_ai/references/atlas_ai_agent_tools#analysis-tools) to your Atlas AI agent. ## Select a runtime version New agents automatically use the latest stable runtime version. You can select a different version in the agent builder user interface (UI) or through the Cognite API. The agent builder displays the runtime version in the agent configuration. Change the version from the dropdown. When you create or update an agent via the [Cognite API](/cdf/explore/agents), specify the `runtimeVersion` field. If you leave this field empty, the agent defaults to the latest stable version. ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "name": "My Agent", "runtimeVersion": "1.3.0" } ``` ## Upgrade to a new version Complete these steps before you upgrade your agent to a new runtime version: 1. Review the feature differences between your current version and the target version. 2. Test your agent thoroughly in a non-production environment. 3. Verify that any custom tools or integrations work correctly with the new version. Atlas AI automatically upgrades agents to the latest stable version when their runtime version reaches the retirement date. ## Agent limits This table lists the current Atlas AI limits per CDF project. To request an increase, contact [Cognite support](https://cognite.zendesk.com/hc/en-us/requests/new). | | Maximum value | | - | - | | Total number of Atlas AI agents | 30 | | Published Atlas AI agents | 10 | | Tools per Atlas AI agent | 20 | # Agent tools library Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_agent_tools Complete reference of built-in Atlas AI agent tools including query tools, analysis tools, and integration tools for interacting with data and external systems. Atlas AI agents use tools to retrieve data, perform calculations, and interact with external systems. Built-in tools cover the most common operations. You can also define custom tools to extend agent capabilities. The query, time series, and document tools are also available outside CDF. See [Adding Industrial MCP to an AI tool](/cdf/build/industrial_mcp) to use them from an external AI tool that supports MCP. ## Query tools The query tools use [**query features**](/cdf/dm/dm_concepts/dm_querying) to retrieve and explore information from the [Cognite knowledge graph](/cdf/dm/dm_concepts/dm_property_graph). | Tool | Use to | Parameters | | - | - | - | | Query knowledge graph | Retrieve data instances from data models in Cognite Data Fusion (CDF). For high accuracy, configure the tool instructions with additional information on the selected data type. | Specify which **data model**, [**view**](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views), and credentials to use. | | [Query tool](/cdf/atlas_ai/references/atlas_ai_query_tool) (**Public preview**) | Retrieve data instances from Data Modeling views by using **list**, **search**, **aggregate**, and **retrieve** operations. Configure **data models** and **instance spaces** in the Agent builder. | Configure **data models** and **instance spaces**. Credentials follow the agent session. | | Query time series data points | Retrieve data instances for **data points**. Use another tool to identify the relevant time series ID.

    See also: [Time series in data modeling](/cdf/dm/dm_guides/dm_integrate_with_time_series). | | ## Analysis tools | Tool | Use to | Parameters | | - | - | - | | Answer document questions | Get answers to questions from a [**file**](/cdf/dm/dm_reference/dm_core_data_model#file) using semantic search. Use in combination with a query tool that retrieves files.

    See also: [Files in data modeling](/cdf/dm/dm_guides/dm_integrate_files) and [Optimizing data models for AI search](/cdf/dm/dm_guides/dm_best_practices_ai_search). | | | Summarize documents | Summarize the information in specific [**files**](/cdf/dm/dm_reference/dm_core_data_model#file). Use in combination with a query tool that retrieves files.

    See also: [Files in data modeling](/cdf/dm/dm_guides/dm_integrate_files). | | | Analyze time series (**Preview**) | Perform advanced calculations on time series **data points**. Use another tool to identify the relevant time series ID.

    See also: [Time series in data modeling](/cdf/dm/dm_guides/dm_integrate_with_time_series). | | | Examine data semantically
    (**Preview**) | Analyze the semantic meaning of data from previously executed queries using natural language prompts. Use to extract insights, identify patterns, or find specific information within query results. This tool analyzes up to 500 instances with a maximum of 150,000 tokens per request.

    **Note**: Requires an agent runtime version that supports [Query memory](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions#query-memory). | | ## Integration tools The integration tools enable Atlas AI agents to interact with external systems and services. The integration tools are not in [production](/cdf/product_feature_status#production) release status yet and may change. | Tool | Use to | Parameters | | - | - | - | | Call REST API | Call a [Cognite API](/api-reference/concepts/20230101/api-description) endpoint. | Specify the API endpoint to call and the method to use. | | [Run Python code](/cdf/atlas_ai/references/atlas_ai_agent_python_tool) | Run custom Python code, and supply instructions for the code to run. | Specify the Python code to run or select an example script. | | [Call Function](/cdf/atlas_ai/references/atlas_ai_call_function_tool) | Call a deployed [Cognite Function](/cdf/functions/use_functions) to run custom Python code. | Specify the function to call, polling time, and the schema of arguments to pass. | | Sandbox
    (**Private preview**) | Run commands and scripts in an isolated, temporary environment with read access to your CDF project.

    **Note**: Always on for agents on [runtime version 2.0.0](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions) and later. See [About Atlas AI agent sandbox and context management](/cdf/atlas_ai/concepts/atlas_ai_agent_sandbox) for details. | No configurable values. | # Call Function Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_call_function_tool Configure the Call Function tool in the Atlas AI agent builder to call deployed Cognite functions from your agents. The Call Function tool enables Atlas AI agents to call deployed [Cognite Functions](/cdf/functions/use_functions) to run custom Python code. Use this tool when you want your agent to perform calculations, process data, integrate with external systems, or automate custom tasks. The integration tools are not in [production](/cdf/product_feature_status#production) release status yet and may change. ## Required capabilities For the Call Function tool to work, the Atlas AI agent must have permission to call Cognite Functions. Grant these capabilities to the CDF group that the agent's service principal belongs to. | Capability | Required for | | - | - | | `functions:run` | Calling a deployed function | | `sessions:create` | Establishing a session when calling the function | If these capabilities are missing, the agent will fail to call the function. Confirm the capabilities are in place before adding the Call Function tool to your agent. For more information on CDF capabilities and how to configure them, see [Setup and administration](/cdf/functions/admin). ## Configuring the tool When you add the Call Function tool to your agent in the [Agent builder](/cdf/atlas_ai/guides/atlas_ai_agent_building), complete these fields: * **Tool name**: A name that identifies the tool's purpose. * **Tool instructions**: Instructions that tell the agent when and how to use this tool. * **Function name**: The deployed Cognite function that the agent calls. * **Max polling time in seconds**: The maximum time the agent waits for the function to complete. The default is 540 seconds. * **Schema**: A JSON Schema that defines the structure of arguments that the agent passes to the function. Each Call Function tool instance calls one function. To call multiple functions, add a separate Call Function tool to your agent for each function. ## Defining the schema The schema defines the arguments the function accepts. Your agent validates arguments against the schema before it calls the function. ### Schema structure This table describes the JSON schema keywords that you can use to define function arguments. | Keyword | Description | Required | | - | - | - | | `type` | Data type of the root object. Set to `"object"` | Yes | | `properties` | Object defining each parameter, including its type and description | Yes | | `required` | Array of property names that must be provided | No | | `description` | Human-readable text that helps the agent understand the parameter | No | ### Schema example This schema defines arguments for a function that retrieves Cognite Data Fusion (CDF) spaces. The `items` property accepts an array of objects that each contain a `space` identifier: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "type": "object", "properties": { "items": { "type": "array", "description": "List of space-ids for the spaces to return.", "items": { "type": "object", "properties": { "space": { "type": "string", "description": "Ids for the spaces to perform operations on" } } } } } } ``` ## Use via API or SDK When you call an agent with this tool enabled through the REST API or Python SDK, the agent will request confirmation before executing the function. Your application receives a `toolConfirmation` action in the response and must reply with `ALLOW` to proceed or `DENY` to cancel. See [Use Atlas AI agents via API and SDK](/cdf/atlas_ai/guides/atlas_ai_agent_api_sdk) for a complete example. ## Related information * [Use functions](/cdf/functions/use_functions) * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) * [Use Atlas AI agents via API and SDK](/cdf/atlas_ai/guides/atlas_ai_agent_api_sdk) # Query tool Source: https://docs.cognite.com/cdf/atlas_ai/references/atlas_ai_query_tool Configure the Query tool in the Atlas AI agent builder so agents can list, search, aggregate, and retrieve data modeling instances. The Query tool enables Atlas AI agents to discover and interact with [data modeling](/cdf/dm/dm_concepts) instances across your entire data model. Agents can list, search, aggregate, or retrieve [instances](/cdf/dm/dm_concepts/dm_spaces_instances#space) using a single, schema-aware tool that works across multiple data models and views. By default, the tool uses the data models and instance spaces for the [location filter](/cdf/locations) you select in the project, so you can switch location filters without updating your agent configuration. **Public preview:** The Query tool is currently in [public preview](/cdf/product_feature_status) testing and is subject to change. ## Configuring the tool When you add the Query tool to your agent in the [Agent builder](/cdf/atlas_ai/guides/atlas_ai_agent_building), configure these settings. | Setting | Description | | - | - | | Tool name | A name that identifies the tool's purpose. | | Tool instructions | Instructions that tell the agent when to use this tool and how to choose data modeling views, filters, and limits. | You must also configure data models and instance spaces. ### Data models The data models setting controls which [data models](/cdf/dm/dm_concepts/dm_containers_views_datamodels#data-models) the agent can query. | Setting | Description | | - | - | | Defined by user's location | The tool uses the data models for your selected location filter. This is the default. | | Configure manually | You provide a fixed list of data models. For each model, you can fill in the **Views** fields to limit discovery to specific [views](/cdf/dm/dm_concepts/dm_containers_views_datamodels#views). If you leave **Views** empty, discovery includes every view in that data model. | ### Instance spaces [Spaces](/cdf/dm/dm_concepts/dm_spaces_instances#space) organize data into separate, access-controlled partitions. The instance spaces setting controls which spaces the Query tool can read from. | Setting | Description | | - | - | | Inherit scope based on user's location | The tool reads from the instance spaces for your selected location filter. This is the default. | | Inherit scope based on user's access rights in CDF | The tool reads from every space you can access in CDF. | | Follow access scope defined manually | You specify exactly which spaces the tool can read from. | ## Operations This table describes the operations available in the Query tool. | Operation | What it does | | - | - | | `list_views` | Discovers which data modeling views the agent can query. | | `get_view_schema` | Returns the fields and relations a data modeling view contains. | | `get_filter_docs` | Returns help text for writing correct filters. | | `list_instances` | Retrieves instances from one data modeling view. | | `search_instances` | Finds instances in one data modeling view that match a text query. | | `aggregate_instances` | Counts and summarizes data in one data modeling view without listing every instance. | | `retrieve_instances` | Fetches specific instances when the agent already knows the instance space and external ID. | ## Example queries This table shows how different question types map to Query tool operations. Understanding this mapping helps you write skills and instructions that guide the agent to the right operation. | Question type | Operation | Example | | - | - | - | | List assets or equipment matching a condition | `list_instances` | "Show me all pumps with status OPEN in Unit 4" | | Find records by name or description text | `search_instances` | "Find work orders mentioning 'bearing failure'" | | Count or summarize records | `aggregate_instances` | "How many open work orders are there per equipment type?" | | Fetch a specific record you already know | `retrieve_instances` | Follow-up retrieval after identifying an instance ID | | Discover what views are available | `list_views` | Used by the agent before querying an unfamiliar data model | | Inspect what fields a view contains | `get_view_schema` | Used by the agent to confirm field names before filtering | ### Filtering, aggregation, and sorting The Query tool supports server-side filtering, aggregation, and sorting. Configuring your skills to use these capabilities keeps queries fast and results focused. Use `aggregate_instances` when a question asks for a count, total, or grouped summary. For example, use a server-side aggregation to answer "How many overdue work orders are there by site?" rather than listing all work orders and counting them. Skills can reinforce this by including a vocabulary mapping: when the user asks "how many," use `aggregate_instances`. Use filters to narrow results before returning them. A question like "Show me all high-priority work orders from the last 30 days" should apply both a priority filter and a date filter in the query rather than retrieving all records and filtering afterward. ### Traversing related data To answer questions that span multiple views, such as finding the time series linked to an asset or the work orders associated with a piece of equipment, the agent uses multiple tool calls. The first call retrieves the primary record. Subsequent calls use IDs from that result to fetch related instances. Document this traversal pattern in a skill when it applies to a common question type, so the agent does not need to infer the correct sequence at runtime. ## Combining tools To complete tasks that include multiple data types, use the Query tool with other tools from the [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools). | Tool | When to use | | - | - | | [Query time series data points](/cdf/atlas_ai/references/atlas_ai_agent_tools#query-tools) | Use the Query tool to identify a time series, then pass its ID to this tool to retrieve data points. | | [Answer document questions](/cdf/atlas_ai/references/atlas_ai_agent_tools#analysis-tools) | Use the Query tool to find and retrieve files, then use this tool to answer questions from their contents. | | [Summarize documents](/cdf/atlas_ai/references/atlas_ai_agent_tools#analysis-tools) | Use the Query tool to retrieve specific files, then use this tool to generate a summary of their contents. | If you are using multiple Query knowledge graph tools, the Query tool can replace them. The Query tool works across multiple data models and views, so you can consolidate your setup, move any view-specific instructions into a [skill](/cdf/atlas_ai/guides/atlas_ai_agent_configure_skills), and use the [evaluation feature](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) to confirm behavior after reconfiguring. ## Related information * [Building agents](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Manage locations](/cdf/locations) * [Agent tools library](/cdf/atlas_ai/references/atlas_ai_agent_tools) * [Querying in data modeling](/cdf/dm/dm_concepts/dm_querying) * [Cognite knowledge graph](/cdf/dm/dm_concepts/dm_property_graph) * [Agent runtime versions](/cdf/atlas_ai/references/atlas_ai_agent_runtime_versions) # Looking up Atlas AI reference information Source: https://docs.cognite.com/cdf/atlas_ai/references/index Look up language models, agent tools, runtime versions, and project limits for Atlas AI agents. Available models, release dates, and retirement timelines by cloud region. Browse all built-in tools by category with configuration options for each. Configuration reference for listing, searching, aggregating, and retrieving instances. Supported parameter types, code structure, and execution environment limits. Connect an agent to a deployed Cognite Function. Runtime versions, feature support, upgrade guidance, and project limits. # Color Source: https://docs.cognite.com/cdf/aura/foundations/color How Aura's two-layer color architecture works — ramps, semantic tokens, light and dark theming, fixed-appearance tokens — plus the complete token and ramp value tables. Aura's color system is built in two layers. **Color ramps** are the raw palette — sequential scales of tints and shades for each hue. **Tokens** are semantic names that map to a specific ramp step and swap their value between light and dark mode automatically. This separation is what makes theming work. When a user switches themes, every token resolves to a different ramp step — but all product code stays untouched, because it only ever references token names, never raw values. The sections below explain how the system is organized; the [theme token](#theme-tokens) and [color ramp](#color-ramps) tables list every resolved value. To extend or customize color tokens in a consuming app, see [Token customization](/cdf/aura/developer-guide/customization). ## The two layers ### Layer 1 — Color ramps Ramps are the source material. Each hue has a sequential scale (for example `--fjord-50` through `--fjord-950`) covering the full range of tints and shades. Alpha variants (`--fjord-alpha-50`, `--fjord-alpha-200`) are semi-transparent overlays built from the ramp's base color, primarily intended for charts and data visualization. Ramps exist so tokens have something to point to. **You should rarely reference ramp values directly in product UI.** The only valid exceptions are: * A color must stay visually constant across both themes (use a fixed token instead if one exists) * You are building a custom marketing or illustration surface where no semantic token fits and theming is not a concern ### Layer 2 — Tokens Tokens are named CSS custom properties that resolve to a ramp step. They are the vocabulary product UI speaks. A token like `--background` points to `--mountain-white` in light mode and `--mountain-950` in dark mode — the name stays the same, the value changes with the theme. ## Light and dark theming Every token has a light mode value and a dark mode value. The token name never changes — only what it resolves to. This means: * Product code never needs `dark:` overrides for tokens * Switching themes is handled entirely by the system * Contrast has been verified for both themes for every token ### How ramp steps map across themes As a general pattern, light mode uses lighter ramp steps for surfaces and darker steps for text. Dark mode inverts this. ## Fixed tokens Fixed tokens are the deliberate exception to the theming rule. They resolve to the **same value in both light and dark mode** — they do not flip with the theme. Unlike theme-adaptive surfaces, fixed colors stay consistent across light and dark mode — keeping these elements from becoming too stark against an inverted background. Fixed tokens are not a workaround for theming. If you find yourself reaching for them outside of persistent chrome, it is a sign the design needs revisiting. Overusing fixed tokens breaks the light/dark contract for users. ## Dos and don'ts Token names reflect their intended use and have been contrast-tested in both themes. Using `--destructive-foreground` for something that isn't destructive, or `--info-background` as a decorative accent, undermines both the visual language and accessibility guarantees. Raw hex values don't respond to theme changes. A hardcoded `#FFFFFF` looks correct in light mode and breaks in dark mode. Always use a token. Manual opacity adjustments (`opacity: 0.5`, custom `rgba()` values) bypass the token system and produce colors that aren't theme-tested. Use alpha tokens from the ramp instead. Semantic tokens — info, success, warning, destructive — exist to communicate system states. Using a green semantic background as a decorative accent trains users to ignore real success states. Use decorative tokens for visual differentiation that carries no status meaning. ## Theme tokens Theme tokens are organized by semantic role. Each token uses a `--group-property` naming convention and resolves to different values in light and dark mode automatically. ### Base The default neutral layer. Used for page backgrounds, primary text, disabled states, and overlays. Start here for any neutral UI. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--base-background` | `#FFFFFF` | | The main background of the app. The lowest layer of the UI. | | `--base-background-hover` | `#F2F3F4` | | Applied as the hover background for interactive elements that have no background of their own by default (like ghost buttons or list rows) and sit directly on the base background. | | `--base-background-hover-fixed-dark` | `#33373A` | | Applied as the hover background for interactive elements that are locked to dark mode, have no background of their own by default (like ghost buttons or list rows), and sit directly on the base background. | | `--base-background-disabled` | `#F2F3F4` | | Used for all disabled interactive elements with backgrounds. | | `--base-background-disabled-fixed-dark` | `#33373A` | | Applied as the disabled background for interactive elements that have a background and are locked to dark mode, and sit directly on the base background. | | `--base-background-overlay` | `#7C868E8C` | | Background between dialogs and UI to help focus attention on the dialog. | | `--base-foreground` | `#17191A` | | The default color for text and icons sitting on base-background or when you need strong contrast and visual hierarchy. | | `--base-foreground-contrast` | `#F8F9F9` | | Foreground-contrast provides a text/icon color that flips between black and white as needed to stay legible against whatever background it's paired with, regardless of light or dark mode. | | `--base-foreground-disabled` | `#B7BCC1` | | The disabled color for text and icons, regardless of background. | | `--base-foreground-disabled-fixed-dark` | `#5C6369` | | Applied as the disabled foreground color for elements that are locked to dark mode, regardless of their background. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--base-background` | `#17191A` | | The main background of the app. The lowest layer of the UI. | | `--base-background-hover` | `#26292C` | | Applied as the hover background for interactive elements that have no background of their own by default (like ghost buttons or list rows) and sit directly on the base background. | | `--base-background-hover-fixed-dark` | `#33373A` | | Applied as the hover background for interactive elements that are locked to dark mode, have no background of their own by default (like ghost buttons or list rows), and sit directly on the base background. | | `--base-background-disabled` | `#26292C` | | Used for all disabled interactive elements with backgrounds. | | `--base-background-disabled-fixed-dark` | `#33373A` | | Applied as the disabled background for interactive elements that have a background and are locked to dark mode, and sit directly on the base background. | | `--base-background-overlay` | `#7C868E8C` | | Background between dialogs and UI to help focus attention on the dialog. | | `--base-foreground` | `#F8F9F9` | | The default color for text and icons sitting on base-background or when you need strong contrast and visual hierarchy. | | `--base-foreground-contrast` | `#17191A` | | Foreground-contrast provides a text/icon color that flips between black and white as needed to stay legible against whatever background it's paired with, regardless of light or dark mode. | | `--base-foreground-disabled` | `#5C6369` | | The disabled color for text and icons, regardless of background. | | `--base-foreground-disabled-fixed-dark` | `#5C6369` | | Applied as the disabled foreground color for elements that are locked to dark mode, regardless of their background. | ### Muted One step of contrast from Base — slightly darker in light mode, slightly lighter in dark mode. Used for subtle containers, sidebars, and shell elements that need separation from the page. For text, used for the lowest-priority content: metadata, captions, timestamps. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--muted-background` | `#F8F9F9` | | The third option for UI background layers, but more commonly used for containers or interactive elements where minimal visual difference is needed or persistent UI shell. | | `--muted-background-hover` | `#F2F3F4` | | Hover state background for muted-background elements. | | `--muted-foreground` | `#6C757C` | | Softest foreground color used for low emphasis such as metadata or foreground elements in inactive states. | | `--muted-foreground-hover` | `#33373A` | | Used for foreground elements using muted-foreground that have an interactive state without a background. | | `--muted-foreground-fixed-dark` | `#A3AAB0` | | Softest foreground color used for low emphasis such as metadata or foreground elements in inactive states but locked to dark mode. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--muted-background` | `#1E2022` | | The third option for UI background layers, but more commonly used for containers or interactive elements where minimal visual difference is needed or persistent UI shell. | | `--muted-background-hover` | `#26292C` | | Hover state background for muted-background elements. | | `--muted-foreground` | `#A3AAB0` | | Softest foreground color used for low emphasis such as metadata or foreground elements in inactive states. | | `--muted-foreground-hover` | `#E2E4E6` | | Used for foreground elements using muted-foreground that have an interactive state without a background. | | `--muted-foreground-fixed-dark` | `#A3AAB0` | | Softest foreground color used for low emphasis such as metadata or foreground elements in inactive states but locked to dark mode. | ### Medium A further step of contrast from Muted. Used for inputs, fields, and interactive surfaces that need to read clearly on top of the page. For text, used for secondary content such as descriptions and long-form body copy. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--medium-background` | `#F2F3F4` | | An alternate to base-background. Ideal for where you want to create some visual difference or interactive elements that sit on top of base-background and need a bit of contrast. Commonly used for inputs. | | `--medium-background-hover` | `#E2E4E6` | | Hover color for interactive elements that use medium-background. | | `--medium-foreground` | `#33373A` | | A slightly softer text color, often paired with medium-background but can be used in an context when less contrast and emphasis is needed for foreground elements. | | `--medium-foreground-hover` | `#7C868E` | | Hover color for interactive foregrounds using medium-foreground but do not have a background color - mainly used for text only interactions without a background. | | `--medium-foreground-fixed-dark` | `#B7BCC1` | | Applied to medium-foreground elements for elements that are locked to dark mode. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--medium-background` | `#26292C` | | An alternate to base-background. Ideal for where you want to create some visual difference or interactive elements that sit on top of base-background and need a bit of contrast. Commonly used for inputs. | | `--medium-background-hover` | `#33373A` | | Hover color for interactive elements that use medium-background. | | `--medium-foreground` | `#B7BCC1` | | A slightly softer text color, often paired with medium-background but can be used in an context when less contrast and emphasis is needed for foreground elements. | | `--medium-foreground-hover` | `#7C868E` | | Hover color for interactive foregrounds using medium-foreground but do not have a background color - mainly used for text only interactions without a background. | | `--medium-foreground-fixed-dark` | `#B7BCC1` | | Applied to medium-foreground elements for elements that are locked to dark mode. | ### Primary (brand) The primary brand color (black for Cognite). Reserved for emphasis and interaction — main CTAs, and selected/active states in components like checkboxes, tabs, radio buttons, and switches. Do not use for general neutral UI. If you need black or dark gray structurally, use Base, not Primary. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--primary-background` | `#1E2022` | | Primary brand color used on buttons and key actions. It's the most prominent color in the UI, so use it sparingly to draw attention to the most important action on screen. | | `--primary-background-hover` | `#4C5257` | | Primary brand color used on hover states for buttons and key actions. It's the most prominent color in the UI, so use it sparingly to draw attention to the most important action on screen. | | `--primary-background-fixed-dark` | `#F8F9F9` | | Used when primary background elements need to remain light in both light and dark mode; paired with primary-background-fixed-dark-hover & primary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--primary-background-hover-fixed-dark` | `#E2E4E6` | | Used when primary background elements need to remain light in both light and dark mode; paired with primary-background-fixed-dark & primary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--primary-foreground` | `#F2F3F4` | | Used for text and icons on top of primary-background. | | `--primary-foreground-fixed-dark` | `#17191A` | | Used when primary text and icon elements need to remain dark in both light and dark mode; paired with primary-background-fixed-dark & primary-background-fixed-dark-hover. This is a fixed dark mode theme. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--primary-background` | `#F8F9F9` | | Primary brand color used on buttons and key actions. It's the most prominent color in the UI, so use it sparingly to draw attention to the most important action on screen. | | `--primary-background-hover` | `#E2E4E6` | | Primary brand color used on hover states for buttons and key actions. It's the most prominent color in the UI, so use it sparingly to draw attention to the most important action on screen. | | `--primary-background-fixed-dark` | `#F8F9F9` | | Used when primary background elements need to remain light in both light and dark mode; paired with primary-background-fixed-dark-hover & primary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--primary-background-hover-fixed-dark` | `#E2E4E6` | | Used when primary background elements need to remain light in both light and dark mode; paired with primary-background-fixed-dark & primary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--primary-foreground` | `#17191A` | | Used for text and icons on top of primary-background. | | `--primary-foreground-fixed-dark` | `#17191A` | | Used when primary text and icon elements need to remain dark in both light and dark mode; paired with primary-background-fixed-dark & primary-background-fixed-dark-hover. This is a fixed dark mode theme. | ### Secondary (brand) The secondary brand color (gray for Cognite). Use for secondary CTAs and toggled/selected states — toggle buttons, selected list items, selected cards. Same rule as Primary: don’t use it for general layout grays; use Muted or Medium for that. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--secondary-background` | `#E2E4E6` | | Secondary brand color (recommended to keep neutral or soft color). Used for secondary actions e.g. secondary buttons or switch background. | | `--secondary-background-hover` | `#F2F3F4` | | The hover state for secondary-background | | `--secondary-background-fixed-dark` | `#33373A` | | Used when secondary background elements need to remain dark in both light and dark mode; paired with secondary-background-fixed-dark-hover & secondary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--secondary-background-hover-fixed-dark` | `#5C6369` | | Used when secondary background elements need to remain dark in both light and dark mode; paired with secondary-background-fixed-dark & secondary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--secondary-foreground` | `#33373A` | | Secondary text and icon colors used for subtitles or long descriptions. | | `--secondary-foreground-fixed-dark` | `#D5D8DB` | | Used when secondary text and icon elements need to remain light in both light and dark mode; paired with secondary-background-fixed-dark & secondary-background-fixed-dark-hover. This is a fixed dark mode theme. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--secondary-background` | `#33373A` | | Secondary brand color (recommended to keep neutral or soft color). Used for secondary actions e.g. secondary buttons or switch background. | | `--secondary-background-hover` | `#5C6369` | | The hover state for secondary-background | | `--secondary-background-fixed-dark` | `#33373A` | | Used when secondary background elements need to remain dark in both light and dark mode; paired with secondary-background-fixed-dark-hover & secondary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--secondary-background-hover-fixed-dark` | `#5C6369` | | Used when secondary background elements need to remain dark in both light and dark mode; paired with secondary-background-fixed-dark & secondary-foreground-fixed-dark. This is a fixed dark mode theme. | | `--secondary-foreground` | `#D5D8DB` | | Secondary text and icon colors used for subtitles or long descriptions. | | `--secondary-foreground-fixed-dark` | `#D5D8DB` | | Used when secondary text and icon elements need to remain light in both light and dark mode; paired with secondary-background-fixed-dark & secondary-background-fixed-dark-hover. This is a fixed dark mode theme. | ### Popover Surface and foreground tokens for contained content in popovers, which are floating, elevated layers (menus, tooltips, dialogs). Highest in the stack; pair with shadow tokens. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--popover-background` | `#FFFFFF` | | Used for surfaces that appear highest in the UI layer and have a drop shadow behind, e.g. Dialog, Alert dialog, Popover, | | `--popover-background-hover` | `#F2F3F4` | | Hover state paired with popover-background. | | `--popover-background-fixed-dark` | `#26292C` | | Used when popover background elements need to remain dark in both light and dark mode. | | `--popover-foreground` | `#17191A` | | Used for foreground elements that sit on popover-background. | | `--popover-foreground-fixed-dark` | `#FFFFFF` | | Used for foreground elements that sit on popover-background, but locked to dark mode and paired with popover-background-fixed:dark. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--popover-background` | `#1E2022` | | Used for surfaces that appear highest in the UI layer and have a drop shadow behind, e.g. Dialog, Alert dialog, Popover, | | `--popover-background-hover` | `#26292C` | | Hover state paired with popover-background. | | `--popover-background-fixed-dark` | `#26292C` | | Used when popover background elements need to remain dark in both light and dark mode. | | `--popover-foreground` | `#F2F3F4` | | Used for foreground elements that sit on popover-background. | | `--popover-foreground-fixed-dark` | `#FFFFFF` | | Used for foreground elements that sit on popover-background, but locked to dark mode and paired with popover-background-fixed:dark. | ### Card Surface and foreground tokens for contained content in cards, which are a distinct surface, but not floating on the page. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--card-background` | `#F8F9F9` | | Use for any card or container background sitting on top of the main background. | | `--card-background-hover` | `#F2F3F4` | | Hover state for cards with card-background. | | `--card-foreground` | `#17191A` | | Used for foregrounds that sit on top of card-background. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--card-background` | `#26292C` | | Use for any card or container background sitting on top of the main background. | | `--card-background-hover` | `#33373A` | | Hover state for cards with card-background. | | `--card-foreground` | `#F2F3F4` | | Used for foregrounds that sit on top of card-background. | ### Destructive Used for irreversible or destructive actions. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--destructive-background` | `#F43D5C` | | Background color for destructive button. | | `--destructive-background-hover` | `#D23751` | | Background hover color for destructive button. | | `--destructive-foreground` | `#F8F9F9` | | Foreground color paired with destructive-background. | | `--destructive-foreground-strong` | `#F43D5C` | | Foreground color for destructive text and icon without destructive-background pairing. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--destructive-background` | `#F43D5C` | | Background color for destructive button. | | `--destructive-background-hover` | `#F7778D` | | Background hover color for destructive button. | | `--destructive-foreground` | `#F8F9F9` | | Foreground color paired with destructive-background. | | `--destructive-foreground-strong` | `#F7778D` | | Foreground color for destructive text and icon without destructive-background pairing. | ### Link Used for hyperlinks that link to external pages. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--link-foreground` | `#5F83E2` | | Used for interactive links. | | `--link-foreground-hover` | `#A7BBEF` | | Hover state for interactive links. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--link-foreground` | `#7796E6` | | Used for interactive links. | | `--link-foreground-hover` | `#CCD7F6` | | Hover state for interactive links. | ### Border Dividers and outlines used for page content separation, containers and inputs. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--border-default` | `#E2E4E6` | | Default border color | | `--border-emphasized` | `#D5D8DB` | | Used for emphasize or to show active or hover states. | | `--border-disabled` | `#F2F3F4` | | Used for disabled borders on interactive elements. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--border-default` | `#33373A` | | Default border color | | `--border-emphasized` | `#5C6369` | | Used for emphasize or to show active or hover states. | | `--border-disabled` | `#26292C` | | Used for disabled borders on interactive elements. | ### Ring Focus indicators for keyboard navigation, including validation states. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--ring-default` | `#5F83E2` | | Used for focus states around inputs as the inner border | | `--ring-muted` | `#CCD7F6` | | Used for focus states around inputs as the outer border | | `--ring-destructive-default` | `#F43D5C` | | Used for validation focus states around inputs as the inner border | | `--ring-destructive-muted` | `#FBC1CB` | | Used for validation focus states around inputs as the outer border | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--ring-default` | `#8FA8EB` | | Used for focus states around inputs as the inner border | | `--ring-muted` | `#3C5085` | | Used for focus states around inputs as the outer border | | `--ring-destructive-default` | `#F7778D` | | Used for validation focus states around inputs as the inner border | | `--ring-destructive-muted` | `#8E2A3B` | | Used for validation focus states around inputs as the outer border | ### Success Indicates a positive, stable, or successful state. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--success-background` | `#D6ECDE` | | Background of success status elements; paired with success-foreground. | | `--success-background-hover` | `#B6DEC5` | | Background hover of success status elements. | | `--success-foreground` | `#19703A` | | Used for success status on text and icon; paired with success-background. | | `--success-foreground-strong` | `#1C984A` | | Used for success status on text and icon without a success-background color. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--success-background` | `#132A1D` | | Background of success status elements; paired with success-foreground. | | `--success-background-hover` | `#153D25` | | Background hover of success status elements. | | `--success-foreground` | `#60B780` | | Used for success status on text and icon; paired with success-background. | | `--success-foreground-strong` | `#60B780` | | Used for success status on text and icon without a success-background color. | ### Warning Indicates an abnormal or warning state. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--warning-background` | `#F8E4D1` | | Background of warning status elements; paired with warning-foreground. | | `--warning-background-hover` | `#F3CFAD` | | Background hover of warning status elements. | | `--warning-foreground` | `#9D5006` | | Used for warning status on text and icon; paired with warning-background. | | `--warning-foreground-strong` | `#D96A00` | | Used for warning status on text and icon without a warning-background color. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--warning-background` | `#352210` | | Background of warning status elements; paired with warning-foreground. | | `--warning-background-hover` | `#512E0D` | | Background hover of warning status elements. | | `--warning-foreground` | `#E4974D` | | Used for warning status on text and icon; paired with warning-background. | | `--warning-foreground-strong` | `#E4974D` | | Used for warning status on text and icon without a warning-background color. | ### Critical Indicates a negative, urgent, or critical state. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--critical-background` | `#FDDCE2` | | Background of critical status elements; paired with critical-foreground-strong. | | `--critical-background-hover` | `#FBC1CB` | | Background hover of critical status elements. | | `--critical-foreground` | `#B03046` | | Used for critical status on text and icon; paired with critical-background. | | `--critical-foreground-strong` | `#F43D5C` | | Used for critical status on text and icon without a critical-background color. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--critical-background` | `#3A1A20` | | Background of critical status elements; paired with critical-foreground-strong. | | `--critical-background-hover` | `#5A202A` | | Background hover of critical status elements. | | `--critical-foreground` | `#F7778D` | | Used for critical status on text and icon; paired with critical-background. | | `--critical-foreground-strong` | `#F7778D` | | Used for critical status on text and icon without a critical-background color. | ### Info Indicates a benign informational state such as running, active, published, or in progress. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--info-background` | `#D9EAFB` | | Background of informational status elements; paired with info-foreground. | | `--info-background-hover` | `#BCDAF8` | | Background hover of informational status elements. | | `--info-foreground` | `#2566A8` | | Used for informational status on text and icon; paired with info-background. | | `--info-foreground-strong` | `#2E8AE8` | | Used for informational status on text and icon without an info-background color. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--info-background` | `#162839` | | Background of informational status elements; paired with info-foreground. | | `--info-background-hover` | `#1A3857` | | Background hover of informational status elements. | | `--info-foreground` | `#6DADEF` | | Used for informational status on text and icon; paired with info-background. | | `--info-foreground-strong` | `#6DADEF` | | Used for informational status on text and icon without an info-background color. | ### Neutral Indicates a neutral or archived state. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--neutral-background` | `#E2E4E6` | | Background of neutral status elements; paired with neutral-foreground. | | `--neutral-background-hover` | `#D5D8DB` | | Background hover of neutral status elements. | | `--neutral-foreground` | `#4C5257` | | Used for neutral status on text and icon; paired with neutral-background. | | `--neutral-foreground-strong` | `#6C757C` | | Used for neutral status on text and icon without a neutral-background color. | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--neutral-background` | `#1E2022` | | Background of neutral status elements; paired with neutral-foreground. | | `--neutral-background-hover` | `#26292C` | | Background hover of neutral status elements. | | `--neutral-foreground` | `#A3AAB0` | | Used for neutral status on text and icon; paired with neutral-background. | | `--neutral-foreground-strong` | `#A3AAB0` | | Used for neutral status on text and icon without a neutral-background color. | ### Decorative Decorative colors are primarily used for badges, alerts, and other decorative visual elements. They do not convey semantic meaning, so semantic status colors should be avoided in these contexts. To support easy re-theming, decorative colors are named 1–7 rather than being tied to specific color names such as Blue or Orange. This naming convention keeps them independent of their visual appearance and makes it easier to update themes without renaming tokens. | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--decorative-1-background` | `#CCD7F6` | | Sets the background color of an element | | `--decorative-1-background-hover` | `#A7BBEF` | | Sets the background hover color of an element | | `--decorative-1-foreground` | `#2A3655` | | Sets the foreground color of an element | | `--decorative-2-background` | `#B6DDD4` | | Sets the background color of an element | | `--decorative-2-background-hover` | `#81C5B6` | | Sets the background hover color of an element | | `--decorative-2-foreground` | `#143C34` | | Sets the foreground color of an element | | `--decorative-3-background` | `#D1E1B8` | | Sets the background color of an element | | `--decorative-3-background-hover` | `#AFCC84` | | Sets the background hover color of an element | | `--decorative-3-foreground` | `#2F4017` | | Sets the foreground color of an element | | `--decorative-4-background` | `#E3CFED` | | Sets the background color of an element | | `--decorative-4-background-hover` | `#D0ADE0` | | Sets the background hover color of an element | | `--decorative-4-foreground` | `#422E4D` | | Sets the foreground color of an element | | `--decorative-5-background` | `#F8C3D9` | | Sets the background color of an element | | `--decorative-5-background-hover` | `#F398BD` | | Sets the background hover color of an element | | `--decorative-5-foreground` | `#562238` | | Sets the foreground color of an element | | `--decorative-6-background` | `#EDDDAD` | | Sets the background color of an element | | `--decorative-6-background-hover` | `#E0C573` | | Sets the background hover color of an element | | `--decorative-6-foreground` | `#4B3C0D` | | Sets the foreground color of an element | | `--decorative-7-background` | `#E2E4E6` | | Sets the background color of an element | | `--decorative-7-background-hover` | `#B7BCC1` | | Sets the background hover color of an element | | `--decorative-7-foreground` | `#33373A` | | Sets the foreground color of an element | | Token | Value | Preview | Usage | | :- | :- | :-: | :- | | `--decorative-1-background` | `#A7BBEF` | | Sets the background color of an element | | `--decorative-1-background-hover` | `#CCD7F6` | | Sets the background hover color of an element | | `--decorative-1-foreground` | `#1F2638` | | Sets the foreground color of an element | | `--decorative-2-background` | `#81C5B6` | | Sets the background color of an element | | `--decorative-2-background-hover` | `#B6DDD4` | | Sets the background hover color of an element | | `--decorative-2-foreground` | `#132A26` | | Sets the foreground color of an element | | `--decorative-3-background` | `#AFCC84` | | Sets the background color of an element | | `--decorative-3-background-hover` | `#D1E1B8` | | Sets the background hover color of an element | | `--decorative-3-foreground` | `#222C15` | | Sets the foreground color of an element | | `--decorative-4-background` | `#D0ADE0` | | Sets the background color of an element | | `--decorative-4-background-hover` | `#E3CFED` | | Sets the background hover color of an element | | `--decorative-4-foreground` | `#2C2233` | | Sets the foreground color of an element | | `--decorative-5-background` | `#F398BD` | | Sets the background color of an element | | `--decorative-5-background-hover` | `#F8C3D9` | | Sets the background hover color of an element | | `--decorative-5-foreground` | `#381B28` | | Sets the foreground color of an element | | `--decorative-6-background` | `#E0C573` | | Sets the background color of an element | | `--decorative-6-background-hover` | `#EDDDAD` | | Sets the background hover color of an element | | `--decorative-6-foreground` | `#322A10` | | Sets the foreground color of an element | | `--decorative-7-background` | `#D5D8DB` | | Sets the background color of an element | | `--decorative-7-background-hover` | `#E2E4E6` | | Sets the background hover color of an element | | `--decorative-7-foreground` | `#33373A` | | Sets the foreground color of an element | ### Chart Chart colors are for data visualizations and should be used in numerical order, starting with `--chart-1`, then `--chart-2`, and so on. | Token | Value | Preview | | :- | :- | :-: | | `--chart-gridlines` | `#E2E4E6` | | | `--chart-1-450` | `#5F83E2D9` | | | `--chart-1-400` | `#5F83E2B3` | | | `--chart-1-300` | `#5F83E28C` | | | `--chart-1-200` | `#5F83E252` | | | `--chart-1-100` | `#5F7DEF2E` | | | `--chart-1-50` | `#5F7DEF14` | | | `--chart-2-450` | `#1A967AD9` | | | `--chart-2-400` | `#1A967AB3` | | | `--chart-2-300` | `#1A967A8C` | | | `--chart-2-200` | `#1A967A52` | | | `--chart-2-100` | `#1A967A2E` | | | `--chart-2-50` | `#1A967A12` | | | `--chart-3-450` | `#6EA220D9` | | | `--chart-3-400` | `#6EA220B3` | | | `--chart-3-300` | `#6EA2208C` | | | `--chart-3-200` | `#6EA22052` | | | `--chart-3-100` | `#6EA2202E` | | | `--chart-3-50` | `#6EA22012` | | | `--chart-4-450` | `#A96AC7D9` | | | `--chart-4-400` | `#A96AC7B3` | | | `--chart-4-300` | `#A96AC78C` | | | `--chart-4-200` | `#A96AC752` | | | `--chart-4-100` | `#A96AC72E` | | | `--chart-4-50` | `#B765C412` | | | `--chart-5-450` | `#C69500D9` | | | `--chart-5-400` | `#C69500B3` | | | `--chart-5-300` | `#C695008C` | | | `--chart-5-200` | `#C6950052` | | | `--chart-5-100` | `#C695002E` | | | `--chart-5-50` | `#C6950012` | | | `--chart-success-450` | `#1C984AD9` | | | `--chart-success-400` | `#1C984AB3` | | | `--chart-success-300` | `#1C984A8C` | | | `--chart-success-200` | `#1C984A52` | | | `--chart-success-100` | `#1C984A2E` | | | `--chart-success-50` | `#1C984A12` | | | `--chart-warning-450` | `#D96A00D9` | | | `--chart-warning-400` | `#D96A00B3` | | | `--chart-warning-300` | `#D96A008C` | | | `--chart-warning-200` | `#D96A0052` | | | `--chart-warning-100` | `#D96A002E` | | | `--chart-warning-50` | `#D96A0012` | | | `--chart-destructive-450` | `#F43D5CD9` | | | `--chart-destructive-400` | `#F43D5CB3` | | | `--chart-destructive-300` | `#F43D5C8C` | | | `--chart-destructive-200` | `#F43D5C52` | | | `--chart-destructive-100` | `#F43D5C2E` | | | `--chart-destructive-50` | `#F43D5C12` | | | `--chart-info-450` | `#2E8AE8D9` | | | `--chart-info-400` | `#2E8AE8B3` | | | `--chart-info-300` | `#2E8AE88C` | | | `--chart-info-200` | `#2E8AE852` | | | `--chart-info-100` | `#2E8AE82E` | | | `--chart-info-50` | `#2E8AE812` | | | `--chart-neutral-450` | `#7C868ED9` | | | `--chart-neutral-400` | `#7C868EB3` | | | `--chart-neutral-300` | `#7C868E8C` | | | `--chart-neutral-200` | `#7C868E52` | | | `--chart-neutral-100` | `#7C868E33` | | | `--chart-neutral-50` | `#7C868E1A` | | | Token | Value | Preview | | :- | :- | :-: | | `--chart-gridlines` | `#26292C` | | | `--chart-1-450` | `#5F83E2D9` | | | `--chart-1-400` | `#5F83E2B3` | | | `--chart-1-300` | `#5F83E28C` | | | `--chart-1-200` | `#5F83E252` | | | `--chart-1-100` | `#5F7DEF2E` | | | `--chart-1-50` | `#5F7DEF14` | | | `--chart-2-450` | `#1A967AD9` | | | `--chart-2-400` | `#1A967AB3` | | | `--chart-2-300` | `#1A967A8C` | | | `--chart-2-200` | `#1A967A52` | | | `--chart-2-100` | `#1A967A2E` | | | `--chart-2-50` | `#1A967A12` | | | `--chart-3-450` | `#6EA220D9` | | | `--chart-3-400` | `#6EA220B3` | | | `--chart-3-300` | `#6EA2208C` | | | `--chart-3-200` | `#6EA22052` | | | `--chart-3-100` | `#6EA2202E` | | | `--chart-3-50` | `#6EA22012` | | | `--chart-4-450` | `#A96AC7D9` | | | `--chart-4-400` | `#A96AC7B3` | | | `--chart-4-300` | `#A96AC78C` | | | `--chart-4-200` | `#A96AC752` | | | `--chart-4-100` | `#A96AC72E` | | | `--chart-4-50` | `#B765C412` | | | `--chart-5-450` | `#C69500D9` | | | `--chart-5-400` | `#C69500B3` | | | `--chart-5-300` | `#C695008C` | | | `--chart-5-200` | `#C6950052` | | | `--chart-5-100` | `#C695002E` | | | `--chart-5-50` | `#C6950012` | | | `--chart-success-450` | `#1C984AD9` | | | `--chart-success-400` | `#1C984AB3` | | | `--chart-success-300` | `#1C984A8C` | | | `--chart-success-200` | `#1C984A52` | | | `--chart-success-100` | `#1C984A2E` | | | `--chart-success-50` | `#1C984A12` | | | `--chart-warning-450` | `#D96A00D9` | | | `--chart-warning-400` | `#D96A00B3` | | | `--chart-warning-300` | `#D96A008C` | | | `--chart-warning-200` | `#D96A0052` | | | `--chart-warning-100` | `#D96A002E` | | | `--chart-warning-50` | `#D96A0012` | | | `--chart-destructive-450` | `#F43D5CD9` | | | `--chart-destructive-400` | `#F43D5CB3` | | | `--chart-destructive-300` | `#F43D5C8C` | | | `--chart-destructive-200` | `#F43D5C52` | | | `--chart-destructive-100` | `#F43D5C2E` | | | `--chart-destructive-50` | `#F43D5C12` | | | `--chart-info-450` | `#2E8AE8D9` | | | `--chart-info-400` | `#2E8AE8B3` | | | `--chart-info-300` | `#2E8AE88C` | | | `--chart-info-200` | `#2E8AE852` | | | `--chart-info-100` | `#2E8AE82E` | | | `--chart-info-50` | `#2E8AE812` | | | `--chart-neutral-450` | `#7C868ED9` | | | `--chart-neutral-400` | `#7C868EB3` | | | `--chart-neutral-300` | `#7C868E8C` | | | `--chart-neutral-200` | `#7C868E52` | | | `--chart-neutral-100` | `#7C868E33` | | | `--chart-neutral-50` | `#7C868E1A` | | ### Heat map Heat map colors communicate intensity, magnitude, or density. They use a sequential progression from lower to higher values, helping users quickly identify patterns, concentrations, and outliers in data visualizations. | Token | Value | Preview | | :- | :- | :-: | | `--heat-map-intense` | `#5372C3` | | | `--heat-map-strong` | `#7796E6` | | | `--heat-map-moderate` | `#A7BBEF` | | | `--heat-map-mild` | `#E2E9FA` | | | Token | Value | Preview | | :- | :- | :-: | | `--heat-map-intense` | `#A7BBEF` | | | `--heat-map-strong` | `#7796E6` | | | `--heat-map-moderate` | `#4861A4` | | | `--heat-map-mild` | `#2A3655` | | ### Shadow Shadow colors define the tint and opacity used to create depth and elevation. They provide consistent visual separation between surfaces while maintaining a cohesive appearance across themes. | Token | Value | Preview | | :- | :- | :-: | | `--shadow-sm` | `#0000000A` | | | `--shadow-md` | `#0000000D` | | | `--shadow-lg` | `#0000000F` | | | `--shadow-xl` | `#0000001A` | | | Token | Value | Preview | | :- | :- | :-: | | `--shadow-sm` | `#00000033` | | | `--shadow-md` | `#0000004D` | | | `--shadow-lg` | `#00000066` | | | `--shadow-xl` | `#00000099` | | *** ## Color ramps Color ramps are sequential scales of tints and shades for a single hue (for example, `--mountain-50` through `--mountain-950`). Themed tokens map to specific steps on these ramps and swap their values between light and dark mode — so `--background` points to `--mountain-white` in light mode and `--mountain-950` in dark mode. **Use themed tokens, not raw ramp values** — unless a color must stay constant across both themes, or you are building a custom marketing or illustration surface where no semantic token fits. Alpha values (`--mountain-50`, `--fjord-200`, etc.) are semi-transparent overlays built from the ramp's base color. They are primarily intended for charts and data visualizations and are intentionally more muted. Light theme alphas map from `50–450` with `500` as the base; dark theme alphas map from `600–950` with `550` as the base. ### Mountain Neutral base palette for text, surfaces, and borders. | Token | Value | Preview | | :- | :- | :-: | | `--mountain-white` | `#FFFFFF` | | | `--mountain-10` | `#F9FAFA` | | | `--mountain-50` | `#F1F2F3` | | | `--mountain-100` | `#E4E6E8` | | | `--mountain-200` | `#D4D7D9` | | | `--mountain-300` | `#BBC0C4` | | | `--mountain-400` | `#A5ABB1` | | | `--mountain-450` | `#8B949A` | | | `--mountain-500` | `#7C868E` | | | `--mountain-550` | `#6D767E` | | | `--mountain-600` | `#5E666D` | | | `--mountain-650` | `#52595F` | | | `--mountain-700` | `#40464A` | | | `--mountain-800` | `#2D3134` | | | `--mountain-900` | `#212426` | | | `--mountain-950` | `#191B1D` | | | `--mountain-black` | `#111213` | | | Token | Value | Preview | | :- | :- | :-: | | `--mountain-50` | `rgba(124, 134, 142, 0.1)` | | | `--mountain-100` | `rgba(124, 134, 142, 0.2)` | | | `--mountain-200` | `rgba(124, 134, 142, 0.32)` | | | `--mountain-300` | `rgba(124, 134, 142, 0.5)` | | | `--mountain-400` | `rgba(124, 134, 142, 0.68)` | | | `--mountain-450` | `rgba(124, 134, 142, 0.88)` | | | `--mountain-600` | `rgba(109, 118, 126, 0.8)` | | | `--mountain-650` | `rgba(109, 118, 126, 0.7)` | | | `--mountain-700` | `rgba(109, 118, 126, 0.55)` | | | `--mountain-800` | `rgba(109, 118, 126, 0.4)` | | | `--mountain-900` | `rgba(109, 118, 126, 0.25)` | | | `--mountain-950` | `rgba(109, 118, 126, 0.1)` | | ### Fjord Primary decorative color used for charts. | Token | Value | Preview | | :- | :- | :-: | | `--fjord-50` | `#EFF2FD` | | | `--fjord-100` | `#DFE5FC` | | | `--fjord-200` | `#CCD5FA` | | | `--fjord-300` | `#AEBDF7` | | | `--fjord-400` | `#93A7F4` | | | `--fjord-450` | `#728DF1` | | | `--fjord-500` | `#5F7DEF` | | | `--fjord-550` | `#486AED` | | | `--fjord-600` | `#2F55EA` | | | `--fjord-650` | `#1742E7` | | | `--fjord-700` | `#1234B6` | | | `--fjord-800` | `#0D2582` | | | `--fjord-900` | `#0A1C61` | | | `--fjord-950` | `#050F33` | | | Token | Value | Preview | | :- | :- | :-: | | `--fjord-50` | `rgba(95, 125, 239, 0.1)` | | | `--fjord-100` | `rgba(95, 125, 239, 0.2)` | | | `--fjord-200` | `rgba(95, 125, 239, 0.32)` | | | `--fjord-300` | `rgba(95, 125, 239, 0.5)` | | | `--fjord-400` | `rgba(95, 125, 239, 0.68)` | | | `--fjord-450` | `rgba(95, 125, 239, 0.88)` | | | `--fjord-600` | `rgba(72, 106, 237, 0.8)` | | | `--fjord-650` | `rgba(72, 106, 237, 0.7)` | | | `--fjord-700` | `rgba(72, 106, 237, 0.55)` | | | `--fjord-800` | `rgba(72, 106, 237, 0.4)` | | | `--fjord-900` | `rgba(72, 106, 237, 0.25)` | | | `--fjord-950` | `rgba(72, 106, 237, 0.1)` | | ### Nordic Primary decorative color used for charts and Atlas AI. | Token | Value | Preview | | :- | :- | :-: | | `--nordic-50` | `#DAF9F2` | | | `--nordic-100` | `#B4F2E5` | | | `--nordic-200` | `#7FE9D2` | | | `--nordic-300` | `#25D7B0` | | | `--nordic-400` | `#21C09D` | | | `--nordic-450` | `#1CA587` | | | `--nordic-500` | `#1A967A` | | | `--nordic-550` | `#17846C` | | | `--nordic-600` | `#14735E` | | | `--nordic-650` | `#116452` | | | `--nordic-700` | `#0D4E40` | | | `--nordic-800` | `#0A372D` | | | `--nordic-900` | `#072922` | | | `--nordic-950` | `#041511` | | | Token | Value | Preview | | :- | :- | :-: | | `--nordic-50` | `rgba(26, 150, 122, 0.1)` | | | `--nordic-100` | `rgba(26, 150, 122, 0.2)` | | | `--nordic-200` | `rgba(26, 150, 122, 0.32)` | | | `--nordic-300` | `rgba(26, 150, 122, 0.5)` | | | `--nordic-400` | `rgba(26, 150, 122, 0.68)` | | | `--nordic-450` | `rgba(26, 150, 122, 0.88)` | | | `--nordic-600` | `rgba(23, 132, 108, 0.8)` | | | `--nordic-650` | `rgba(23, 132, 108, 0.7)` | | | `--nordic-700` | `rgba(23, 132, 108, 0.55)` | | | `--nordic-800` | `rgba(23, 132, 108, 0.4)` | | | `--nordic-900` | `rgba(23, 132, 108, 0.25)` | | | `--nordic-950` | `rgba(23, 132, 108, 0.1)` | | ### Aurora Primary decorative color used for charts and decorative elements. | Token | Value | Preview | | :- | :- | :-: | | `--aurora-50` | `#E7F9C6` | | | `--aurora-100` | `#CFF38D` | | | `--aurora-200` | `#ABE938` | | | `--aurora-300` | `#90D217` | | | `--aurora-400` | `#81BC14` | | | `--aurora-450` | `#6EA112` | | | `--aurora-500` | `#649210` | | | `--aurora-550` | `#58810E` | | | `--aurora-600` | `#4D700C` | | | `--aurora-650` | `#43610B` | | | `--aurora-700` | `#344C08` | | | `--aurora-800` | `#253606` | | | `--aurora-900` | `#1B2804` | | | `--aurora-950` | `#0E1402` | | | Token | Value | Preview | | :- | :- | :-: | | `--aurora-50` | `rgba(100, 146, 16, 0.1)` | | | `--aurora-100` | `rgba(100, 146, 16, 0.2)` | | | `--aurora-200` | `rgba(100, 146, 16, 0.32)` | | | `--aurora-300` | `rgba(100, 146, 16, 0.5)` | | | `--aurora-400` | `rgba(100, 146, 16, 0.68)` | | | `--aurora-450` | `rgba(100, 146, 16, 0.88)` | | | `--aurora-600` | `rgba(88, 129, 14, 0.8)` | | | `--aurora-650` | `rgba(88, 129, 14, 0.7)` | | | `--aurora-700` | `rgba(88, 129, 14, 0.55)` | | | `--aurora-800` | `rgba(88, 129, 14, 0.4)` | | | `--aurora-900` | `rgba(88, 129, 14, 0.25)` | | | `--aurora-950` | `rgba(88, 129, 14, 0.1)` | | ### Dusk Primary decorative color used for charts and decorative elements. | Token | Value | Preview | | :- | :- | :-: | | `--dusk-50` | `#F8EFF9` | | | `--dusk-100` | `#F1E1F3` | | | `--dusk-200` | `#E8CEEC` | | | `--dusk-300` | `#DBB1E1` | | | `--dusk-400` | `#CF97D7` | | | `--dusk-450` | `#C078CB` | | | `--dusk-500` | `#B765C4` | | | `--dusk-550` | `#AD4EBB` | | | `--dusk-600` | `#9940A6` | | | `--dusk-650` | `#853790` | | | `--dusk-700` | `#682C71` | | | `--dusk-800` | `#4A1F50` | | | `--dusk-900` | `#37173C` | | | `--dusk-950` | `#1C0C1F` | | | Token | Value | Preview | | :- | :- | :-: | | `--dusk-50` | `rgba(183, 101, 196, 0.1)` | | | `--dusk-100` | `rgba(183, 101, 196, 0.2)` | | | `--dusk-200` | `rgba(183, 101, 196, 0.32)` | | | `--dusk-300` | `rgba(183, 101, 196, 0.5)` | | | `--dusk-400` | `rgba(183, 101, 196, 0.68)` | | | `--dusk-450` | `rgba(183, 101, 196, 0.88)` | | | `--dusk-600` | `rgba(173, 78, 187, 0.8)` | | | `--dusk-650` | `rgba(173, 78, 187, 0.7)` | | | `--dusk-700` | `rgba(173, 78, 187, 0.55)` | | | `--dusk-800` | `rgba(173, 78, 187, 0.4)` | | | `--dusk-900` | `rgba(173, 78, 187, 0.25)` | | | `--dusk-950` | `rgba(173, 78, 187, 0.1)` | | ### Sky Secondary decorative color used for badges, avatars, and illustrations. | Token | Value | Preview | | :- | :- | :-: | | `--sky-50` | `#DBF7FF` | | | `--sky-100` | `#B7EFFE` | | | `--sky-200` | `#84E4FE` | | | `--sky-300` | `#25CFFD` | | | `--sky-400` | `#02B9EA` | | | `--sky-450` | `#029FC9` | | | `--sky-500` | `#0290B6` | | | `--sky-550` | `#017FA1` | | | `--sky-600` | `#016E8C` | | | `--sky-650` | `#01607A` | | | `--sky-700` | `#014B60` | | | `--sky-800` | `#013644` | | | `--sky-900` | `#002832` | | | `--sky-950` | `#00151A` | | | Token | Value | Preview | | :- | :- | :-: | | `--sky-50` | `rgba(2, 144, 182, 0.1)` | | | `--sky-100` | `rgba(2, 144, 182, 0.2)` | | | `--sky-200` | `rgba(2, 144, 182, 0.32)` | | | `--sky-300` | `rgba(2, 144, 182, 0.5)` | | | `--sky-400` | `rgba(2, 144, 182, 0.68)` | | | `--sky-450` | `rgba(2, 144, 182, 0.88)` | | | `--sky-600` | `rgba(1, 127, 161, 0.8)` | | | `--sky-650` | `rgba(1, 127, 161, 0.7)` | | | `--sky-700` | `rgba(1, 127, 161, 0.55)` | | | `--sky-800` | `rgba(1, 127, 161, 0.4)` | | | `--sky-900` | `rgba(1, 127, 161, 0.25)` | | | `--sky-950` | `rgba(1, 127, 161, 0.1)` | | ### Orange Secondary decorative color used for badges, avatars, and illustrations. | Token | Value | Preview | | :- | :- | :-: | | `--orange-50` | `#FEEFE6` | | | `--orange-100` | `#FDE0CF` | | | `--orange-200` | `#FCCDB0` | | | `--orange-300` | `#FBAE7E` | | | `--orange-400` | `#F98F4E` | | | `--orange-450` | `#F7640A` | | | `--orange-500` | `#E15A07` | | | `--orange-550` | `#C75006` | | | `--orange-600` | `#AD4506` | | | `--orange-650` | `#963C05` | | | `--orange-700` | `#762F04` | | | `--orange-800` | `#542203` | | | `--orange-900` | `#3E1902` | | | `--orange-950` | `#210D01` | | | Token | Value | Preview | | :- | :- | :-: | | `--orange-50` | `rgba(225, 90, 7, 0.1)` | | | `--orange-100` | `rgba(225, 90, 7, 0.2)` | | | `--orange-200` | `rgba(225, 90, 7, 0.32)` | | | `--orange-300` | `rgba(225, 90, 7, 0.5)` | | | `--orange-400` | `rgba(225, 90, 7, 0.68)` | | | `--orange-450` | `rgba(225, 90, 7, 0.88)` | | | `--orange-600` | `rgba(199, 80, 6, 0.8)` | | | `--orange-650` | `rgba(199, 80, 6, 0.7)` | | | `--orange-700` | `rgba(199, 80, 6, 0.55)` | | | `--orange-800` | `rgba(199, 80, 6, 0.4)` | | | `--orange-900` | `rgba(199, 80, 6, 0.25)` | | | `--orange-950` | `rgba(199, 80, 6, 0.1)` | | ### Yellow Secondary decorative color used for badges, avatars, and illustrations. | Token | Value | Preview | | :- | :- | :-: | | `--yellow-50` | `#FFF4A2` | | | `--yellow-100` | `#FFE834` | | | `--yellow-200` | `#F3D700` | | | `--yellow-300` | `#D8BF00` | | | `--yellow-400` | `#C1AB00` | | | `--yellow-450` | `#A69300` | | | `--yellow-500` | `#968500` | | | `--yellow-550` | `#857600` | | | `--yellow-600` | `#736600` | | | `--yellow-650` | `#645900` | | | `--yellow-700` | `#4E4500` | | | `--yellow-800` | `#373100` | | | `--yellow-900` | `#292400` | | | `--yellow-950` | `#151200` | | | Token | Value | Preview | | :- | :- | :-: | | `--yellow-50` | `rgba(150, 133, 0, 0.1)` | | | `--yellow-100` | `rgba(150, 133, 0, 0.2)` | | | `--yellow-200` | `rgba(150, 133, 0, 0.32)` | | | `--yellow-300` | `rgba(150, 133, 0, 0.5)` | | | `--yellow-400` | `rgba(150, 133, 0, 0.68)` | | | `--yellow-450` | `rgba(150, 133, 0, 0.88)` | | | `--yellow-600` | `rgba(133, 118, 0, 0.8)` | | | `--yellow-650` | `rgba(133, 118, 0, 0.7)` | | | `--yellow-700` | `rgba(133, 118, 0, 0.55)` | | | `--yellow-800` | `rgba(133, 118, 0, 0.4)` | | | `--yellow-900` | `rgba(133, 118, 0, 0.25)` | | | `--yellow-950` | `rgba(133, 118, 0, 0.1)` | | ### Pink Secondary decorative color used for badges, avatars, and illustrations. | Token | Value | Preview | | :- | :- | :-: | | `--pink-50` | `#FDEEF4` | | | `--pink-100` | `#FBDEEA` | | | `--pink-200` | `#F9CADD` | | | `--pink-300` | `#F5A9C8` | | | `--pink-400` | `#F28BB4` | | | `--pink-450` | `#ED6199` | | | `--pink-500` | `#E94487` | | | `--pink-550` | `#E11A6A` | | | `--pink-600` | `#C4175C` | | | `--pink-650` | `#AB1451` | | | `--pink-700` | `#881040` | | | `--pink-800` | `#620B2E` | | | `--pink-900` | `#490823` | | | `--pink-950` | `#280513` | | | Token | Value | Preview | | :- | :- | :-: | | `--pink-50` | `rgba(233, 68, 135, 0.1)` | | | `--pink-100` | `rgba(233, 68, 135, 0.2)` | | | `--pink-200` | `rgba(233, 68, 135, 0.32)` | | | `--pink-300` | `rgba(233, 68, 135, 0.5)` | | | `--pink-400` | `rgba(233, 68, 135, 0.68)` | | | `--pink-450` | `rgba(233, 68, 135, 0.88)` | | | `--pink-600` | `rgba(225, 26, 106, 0.8)` | | | `--pink-650` | `rgba(225, 26, 106, 0.7)` | | | `--pink-700` | `rgba(225, 26, 106, 0.55)` | | | `--pink-800` | `rgba(225, 26, 106, 0.4)` | | | `--pink-900` | `rgba(225, 26, 106, 0.25)` | | | `--pink-950` | `rgba(225, 26, 106, 0.1)` | | ### Gray Used for focus states and secondary decorative elements. | Token | Value | Preview | | :- | :- | :-: | | `--gray-50` | `#F0F2F9` | | | `--gray-100` | `#E2E5F4` | | | `--gray-200` | `#D0D6ED` | | | `--gray-300` | `#B5BEE2` | | | `--gray-400` | `#9DA9D9` | | | `--gray-450` | `#818FCE` | | | `--gray-500` | `#7081C7` | | | `--gray-550` | `#5D70C0` | | | `--gray-600` | `#4A5FB8` | | | `--gray-650` | `#3F52A3` | | | `--gray-700` | `#32417F` | | | `--gray-800` | `#232E5A` | | | `--gray-900` | `#1A2242` | | | `--gray-950` | `#0D1122` | | | Token | Value | Preview | | :- | :- | :-: | | `--gray-50` | `rgba(112, 129, 199, 0.1)` | | | `--gray-100` | `rgba(112, 129, 199, 0.2)` | | | `--gray-200` | `rgba(112, 129, 199, 0.32)` | | | `--gray-300` | `rgba(112, 129, 199, 0.5)` | | | `--gray-400` | `rgba(112, 129, 199, 0.68)` | | | `--gray-450` | `rgba(112, 129, 199, 0.88)` | | | `--gray-600` | `rgba(93, 112, 192, 0.8)` | | | `--gray-650` | `rgba(93, 112, 192, 0.7)` | | | `--gray-700` | `rgba(93, 112, 192, 0.55)` | | | `--gray-800` | `rgba(93, 112, 192, 0.4)` | | | `--gray-900` | `rgba(93, 112, 192, 0.25)` | | | `--gray-950` | `rgba(93, 112, 192, 0.1)` | | ### Red Maps to semantic status colors for destructive actions and critical alerts. | Token | Value | Preview | | :- | :- | :-: | | `--red-50` | `#FEEEF1` | | | `--red-100` | `#FDDEE4` | | | `--red-200` | `#FCCAD2` | | | `--red-300` | `#FAA9B7` | | | `--red-400` | `#F8899C` | | | `--red-450` | `#F65E78` | | | `--red-500` | `#F43D5C` | | | `--red-550` | `#E90D32` | | | `--red-600` | `#CB0B2C` | | | `--red-650` | `#B20A26` | | | `--red-700` | `#8D081F` | | | `--red-800` | `#660616` | | | `--red-900` | `#4E0411` | | | `--red-950` | `#2C0209` | | | Token | Value | Preview | | :- | :- | :-: | | `--red-50` | `rgba(244, 61, 92, 0.1)` | | | `--red-100` | `rgba(244, 61, 92, 0.2)` | | | `--red-200` | `rgba(244, 61, 92, 0.32)` | | | `--red-300` | `rgba(244, 61, 92, 0.5)` | | | `--red-400` | `rgba(244, 61, 92, 0.68)` | | | `--red-450` | `rgba(244, 61, 92, 0.88)` | | | `--red-600` | `rgba(233, 13, 50, 0.8)` | | | `--red-650` | `rgba(233, 13, 50, 0.7)` | | | `--red-700` | `rgba(233, 13, 50, 0.55)` | | | `--red-800` | `rgba(233, 13, 50, 0.4)` | | | `--red-900` | `rgba(233, 13, 50, 0.25)` | | | `--red-950` | `rgba(233, 13, 50, 0.1)` | | ### Amber Maps to semantic status colors for warnings. | Token | Value | Preview | | :- | :- | :-: | | `--amber-50` | `#FFF1D0` | | | `--amber-100` | `#FFE3A2` | | | `--amber-200` | `#FFD062` | | | `--amber-300` | `#FCB100` | | | `--amber-400` | `#E19E00` | | | `--amber-450` | `#C18800` | | | `--amber-500` | `#AF7B00` | | | `--amber-550` | `#9B6D00` | | | `--amber-600` | `#865E00` | | | `--amber-650` | `#755200` | | | `--amber-700` | `#5B4000` | | | `--amber-800` | `#412D00` | | | `--amber-900` | `#302200` | | | `--amber-950` | `#181100` | | | Token | Value | Preview | | :- | :- | :-: | | `--amber-50` | `rgba(175, 123, 0, 0.1)` | | | `--amber-100` | `rgba(175, 123, 0, 0.2)` | | | `--amber-200` | `rgba(175, 123, 0, 0.32)` | | | `--amber-300` | `rgba(175, 123, 0, 0.5)` | | | `--amber-400` | `rgba(175, 123, 0, 0.68)` | | | `--amber-450` | `rgba(175, 123, 0, 0.88)` | | | `--amber-600` | `rgba(155, 109, 0, 0.8)` | | | `--amber-650` | `rgba(155, 109, 0, 0.7)` | | | `--amber-700` | `rgba(155, 109, 0, 0.55)` | | | `--amber-800` | `rgba(155, 109, 0, 0.4)` | | | `--amber-900` | `rgba(155, 109, 0, 0.25)` | | | `--amber-950` | `rgba(155, 109, 0, 0.1)` | | ### Green Maps to semantic status colors for success. | Token | Value | Preview | | :- | :- | :-: | | `--green-50` | `#DDF9E7` | | | `--green-100` | `#BBF3D0` | | | `--green-200` | `#8BEAAE` | | | `--green-300` | `#34DA71` | | | `--green-400` | `#24C45E` | | | `--green-450` | `#1FA851` | | | `--green-500` | `#1C984A` | | | `--green-550` | `#198741` | | | `--green-600` | `#157538` | | | `--green-650` | `#136631` | | | `--green-700` | `#0F5026` | | | `--green-800` | `#0A381B` | | | `--green-900` | `#082A14` | | | `--green-950` | `#04160A` | | | Token | Value | Preview | | :- | :- | :-: | | `--green-50` | `rgba(28, 152, 74, 0.1)` | | | `--green-100` | `rgba(28, 152, 74, 0.2)` | | | `--green-200` | `rgba(28, 152, 74, 0.32)` | | | `--green-300` | `rgba(28, 152, 74, 0.5)` | | | `--green-400` | `rgba(28, 152, 74, 0.68)` | | | `--green-450` | `rgba(28, 152, 74, 0.88)` | | | `--green-600` | `rgba(25, 135, 65, 0.8)` | | | `--green-650` | `rgba(25, 135, 65, 0.7)` | | | `--green-700` | `rgba(25, 135, 65, 0.55)` | | | `--green-800` | `rgba(25, 135, 65, 0.4)` | | | `--green-900` | `rgba(25, 135, 65, 0.25)` | | | `--green-950` | `rgba(25, 135, 65, 0.1)` | | ### Blue Maps to semantic status colors for information. | Token | Value | Preview | | :- | :- | :-: | | `--blue-50` | `#E6F4FF` | | | `--blue-100` | `#CEEAFF` | | | `--blue-200` | `#AEDCFF` | | | `--blue-300` | `#7CC6FF` | | | `--blue-400` | `#4CB2FF` | | | `--blue-450` | `#0B95FF` | | | `--blue-500` | `#0086ED` | | | `--blue-550` | `#0077D2` | | | `--blue-600` | `#0067B6` | | | `--blue-650` | `#005A9F` | | | `--blue-700` | `#00477D` | | | `--blue-800` | `#003259` | | | `--blue-900` | `#002542` | | | `--blue-950` | `#001423` | | | Token | Value | Preview | | :- | :- | :-: | | `--blue-50` | `rgba(0, 134, 237, 0.1)` | | | `--blue-100` | `rgba(0, 134, 237, 0.2)` | | | `--blue-200` | `rgba(0, 134, 237, 0.32)` | | | `--blue-300` | `rgba(0, 134, 237, 0.5)` | | | `--blue-400` | `rgba(0, 134, 237, 0.68)` | | | `--blue-450` | `rgba(0, 134, 237, 0.88)` | | | `--blue-600` | `rgba(0, 119, 210, 0.8)` | | | `--blue-650` | `rgba(0, 119, 210, 0.7)` | | | `--blue-700` | `rgba(0, 119, 210, 0.55)` | | | `--blue-800` | `rgba(0, 119, 210, 0.4)` | | | `--blue-900` | `rgba(0, 119, 210, 0.25)` | | | `--blue-950` | `rgba(0, 119, 210, 0.1)` | | # Effects Source: https://docs.cognite.com/cdf/aura/foundations/effects Effects include elevations (shadows) and focus rings for consistent depth, attention, and keyboard focus. ## Elevations Elevation, or shadow, adds depth and hierarchy to the interface by using shadow to separate elements from the background. It helps guide user focus and indicate layers of interaction. Elevation should be applied intentionally and consistently to avoid visual clutter. ### Guidelines * Use elevation tokens, not hardcoded box-shadow values. * Elevation should reflect actual layering — tooltips and popovers sit higher than cards, which sit higher than the base surface. Don't apply high elevation to flat, inline elements. * Don't use shadows decoratively. If an element isn't functionally elevated (e.g. floating, overlaying content), it shouldn't have a shadow. * Avoid stacking shadows — if multiple elevated elements overlap, only the topmost should cast a shadow. * In dark mode, elevation is often conveyed through surface lightness rather than shadow intensity. Let the tokens handle this automatically. * Avoid shadows on cards, especially in dense dashboard layouts — they add visual noise and make the page feel cluttered. ### Tokens | Token | Use | | :- | :- | | `effect-shadow-sm` | Tooltips, small floating labels | | `effect-shadow-default` | Menus, Dropdowns, Popovers, or floating panels and toolbars | | `effect-shadow-lg` | Drawers, Sheets, Side panels | | `effect-shadow-xl` | Dialogs, Alert dialogs | ## Focus states Focus states, or rings, make interactive elements visible to keyboard and assistive technology users. They are not optional. ### Guidelines * Never remove or suppress focus styles — not even for aesthetic reasons. * Don't rely on color alone to show focus (always use a 2px stroke weight); the focus ring must be visible regardless of the element's color or background. * Focus styles are provided by Aura via primitives and tokens. Don't override them unless you are replacing them with something that meets WCAG 2.1 AA contrast requirements. * Custom interactive elements (anything not using an Aura component) must implement focus styles manually. * Test focus order — it should follow a logical reading sequence and not jump unexpectedly around the page. ### Tokens **Default focus ring** | Token | Value | Description | | :- | :- | :- | | `ring` + `border-width/1` | `1px solid, #7081C7` | Color and stroke weight applied to inside stroke weight of the focus ring | | `ring-muted` + `border-width/1` | `1px solid, #D0D6ED` | Color and stroke weight applied to outside stroke weight of the focus ring | **Validation focus ring** | Token | Value | Description | | :- | :- | :- | | `ring-destructive` + `border-width/1` | `1px solid, #CB0B2C` | Color and stroke weight applied to inside stroke weight of the focus ring | | `ring-destructive-muted` + `border-width/1` | `1px solid, #FDDEE4` | Color and stroke weight applied to outside stroke weight of the focus ring | # Exploring Aura foundations Source: https://docs.cognite.com/cdf/aura/foundations/index Explore Aura foundations from color tokens to spacing to accessibility to understand how primitives are composed. Use these guides to understand tokens, layout rhythm, motion cues, interaction signifiers, heuristic checks, and WCAG-oriented practices that underpin Aura components. Learn how ramps, semantic theme tokens, and chart palettes stay theme-safe across light and dark UI. Apply typefaces, scale, and typography tokens so text stays consistent and readable everywhere. Use spacing, sizing, radius, and stroke tokens to establish hierarchy and rhythm in layouts. Apply elevation and focus-ring tokens for depth, attention, and predictable keyboard focus. Apply easing, duration, and stagger tokens for productive and expressive animation with Tailwind utilities. See how affordances, signifiers, and built-in states keep interactive UI clear and consistent. Evaluate and improve screens with practical usability heuristics aligned to Aura patterns. Meet shared contrast, focus, semantics, and media expectations so interfaces work for more people. Extend or override Aura style tokens in a consuming app when scoped overrides are appropriate. # Size Source: https://docs.cognite.com/cdf/aura/foundations/size Size covers the spatial rules that give a UI its structure and rhythm — height, spacing, radius, and stroke width ## Spacing Spacing is the negative area between elements and components both horizontally and vertically. It is commonly controlled in code with gap, margin or padding. Every part of a UI should be intentional including the empty space between elements. The amount of space between items creates relationships and hierarchy. ### Guidelines #### Creating relationships * Elements in a design that are near each other are seen as being meaningfully related. As more space is added between elements, their perceived relationship weakens. * Space can also be used to denote groups of associated information. This creates content sections on a page without having to use lines or other graphical elements as a divider. #### Creating hierarchy * Elements that have more spacing around them tend to be perceived as higher in importance than elements that have less space around them. * Elements that are set close to each other can be easily overlooked. Users may see the grouping but not process each individual item. Therefore, if you have an element or content of high importance on the page, consider giving it extra surrounding space to help it attract focus. #### White space * Empty space - also known as white space - is important in design. It can be used to break up sections on a page or to help create focus on certain element(s). * White space helps with information processing; too much dense information can be disorienting or overwhelming for a user. Sections of a UI are allowed to be dense, but the whole page should not be crowded; there should always be enough white space to let the user’s eyes rest. The **Utility classes** variable collection in Figma groups sizing utilities by name prefix (`padding/…`, `margin/…`, and so on). Each utility aliases a **spacing**, **radius**, or **border-width** primitive and resolves to a pixel value in the default mode. The tables below list those groupings. ### Tokens #### Padding Applied as top, right, bottom, left padding within a component or UI element. | Utility variable | Aliased token | Value | | :- | :- | :- | | `padding/p-2` | `spacing/2` | 2 px | | `padding/p-4` | `spacing/4` | 4 px | | `padding/p-6` | `spacing/6` | 6 px | | `padding/p-8` | `spacing/8` | 8 px | | `padding/p-12` | `spacing/12` | 12 px | | `padding/p-14` | `spacing/14` | 14 px | | `padding/p-16` | `spacing/16` | 16 px | | `padding/p-20` | `spacing/20` | 20 px | | `padding/p-24` | `spacing/24` | 24 px | | `padding/p-28` | `spacing/28` | 28 px | | `padding/p-32` | `spacing/32` | 32 px | | `padding/p-36` | `spacing/36` | 36 px | | `padding/p-40` | `spacing/40` | 40 px | #### Margin Applied as top, right, bottom, left margin outside of a component or UI element. | Utility variable | Aliased token | Value | | :- | :- | :- | | `margin/m-4` | `spacing/4` | 4 px | | `margin/m-6` | `spacing/6` | 6 px | | `margin/m-8` | `spacing/8` | 8 px | | `margin/m-12` | `spacing/12` | 12 px | | `margin/m-16` | `spacing/16` | 16 px | | `margin/m-20` | `spacing/20` | 20 px | | `margin/m-24` | `spacing/24` | 24 px | | `margin/m-28` | `spacing/28` | 28 px | | `margin/m-32` | `spacing/32` | 32 px | | `margin/m-36` | `spacing/36` | 36 px | | `margin/m-40` | `spacing/40` | 40 px | | `margin/m-44` | `spacing/44` | 44 px | | `margin/m-48` | `spacing/48` | 48 px | | `margin/m-52` | `spacing/52` | 52 px | | `margin/m-56` | `spacing/56` | 56 px | | `margin/m-60` | `spacing/60` | 60 px | | `margin/m-64` | `spacing/64` | 64 px | | `margin/m-68` | `spacing/68` | 68 px | | `margin/m-72` | `spacing/72` | 72 px | | `margin/m-76` | `spacing/76` | 76 px | | `margin/m-80` | `spacing/80` | 80 px | ## Size Size is the vertical height of components and are used to create visual consistency across the system. Use size tokens to setup component's height or as a reference to calculate the final size of the component. Each size has a prefixed height and name for easy reference. For example, our Button component is available in three sizes: small (28px), default - or medium - (36px), and large (40px). ### Guidelines #### Consistency across components * Use the same size components consistently across similar components to maintain alignment, rhythm, and predictable behavior throughout the interface. * This helps prevent visual noise and ensures that mixed components (like inputs and buttons) align cleanly in layouts. #### Touch target best practices * Always aim for a minimum touch target of 40-44px for tappable elements on mobile to meet accessibility guidelines and user comfort. * If this is not feasible in height, ensure the width is wide enough to be easily accessible. #### Variable height * Some components don’t have a fixed height, so they can grow vertically and adapt to the amount of content they hold. In these components, padding is important to align on. #### Corresponding radius * Each size corresponds with a preferred radius value. This is because the size and corner radius are often proportional to one another and should scale accordingly, with the exception of rounded fully corner radii. * For example, default Buttons and Inputs (36px h) always use an 8px corner radius. However, small Butons and Inputs (28px h) use a 6px corner radius. ### Tokens #### Width Applied as a fixed width to a component, e.g., Button. | Utility variable | Aliased token | Value | | :- | :- | :- | | `width/w-4` | `spacing/4` | 4 px | | `width/w-8` | `spacing/8` | 8 px | | `width/w-16` | `spacing/16` | 16 px | | `width/w-20` | `spacing/20` | 20 px | | `width/w-24` | `spacing/24` | 24 px | | `width/w-28` | `spacing/28` | 28 px | | `width/w-32` | `spacing/32` | 32 px | | `width/w-36` | `spacing/36` | 36 px | | `width/w-40` | `spacing/40` | 40 px | | `width/w-44` | `spacing/44` | 44 px | | `width/w-48` | `spacing/48` | 48 px | #### Min-width Applied when a min width is set to a component or container, e.g., Dialog. | Utility variable | Aliased token | Value | | :- | :- | :- | | `min-width/min-w-4` | `spacing/4` | 4 px | | `min-width/min-w-8` | `spacing/8` | 8 px | | `min-width/min-w-16` | `spacing/16` | 16 px | | `min-width/min-w-20` | `spacing/20` | 20 px | | `min-width/min-w-24` | `spacing/24` | 24 px | | `min-width/min-w-28` | `spacing/28` | 28 px | | `min-width/min-w-32` | `spacing/32` | 32 px | | `min-width/min-w-36` | `spacing/36` | 36 px | | `min-width/min-w-40` | `spacing/40` | 40 px | | `min-width/min-w-44` | `spacing/44` | 44 px | | `min-width/min-w-48` | `spacing/48` | 48 px | #### Max-width Applied when a max width is set to a component or container, e.g., Dialog. | Utility variable | Aliased token | Value | | :- | :- | :- | | `max-width/max-w-4` | `spacing/4` | 4 px | | `max-width/max-w-8` | `spacing/8` | 8 px | | `max-width/max-w-16` | `spacing/16` | 16 px | | `max-width/max-w-20` | `spacing/20` | 20 px | | `max-width/max-w-24` | `spacing/24` | 24 px | | `max-width/max-w-28` | `spacing/28` | 28 px | | `max-width/max-w-32` | `spacing/32` | 32 px | | `max-width/max-w-36` | `spacing/36` | 36 px | | `max-width/max-w-40` | `spacing/40` | 40 px | | `max-width/max-w-44` | `spacing/44` | 44 px | | `max-width/max-w-48` | `spacing/48` | 48 px | | `max-width/max-w-52` | `spacing/52` | 52 px | | `max-width/max-w-56` | `spacing/56` | 56 px | | `max-width/max-w-60` | `spacing/60` | 60 px | | `max-width/max-w-64` | `spacing/64` | 64 px | | `max-width/max-w-68` | `spacing/68` | 68 px | | `max-width/max-w-72` | `spacing/72` | 72 px | | `max-width/max-w-76` | `spacing/76` | 76 px | | `max-width/max-w-80` | `spacing/80` | 80 px | | `max-width/max-w-84` | `spacing/84` | 84 px | | `max-width/max-w-88` | `spacing/88` | 88 px | | `max-width/max-w-92` | `spacing/92` | 92 px | | `max-width/max-w-96` | `spacing/96` | 96 px | | `max-width/max-w-100` | `spacing/100` | 100 px | #### Height Applied as a fixed height to a component, e.g., Button. | Utility variable | Aliased token | Value | | :- | :- | :- | | `height/h-4` | `spacing/4` | 4 px | | `height/h-8` | `spacing/8` | 8 px | | `height/h-16` | `spacing/16` | 16 px | | `height/h-20` | `spacing/20` | 20 px | | `height/h-24` | `spacing/24` | 24 px | | `height/h-28` | `spacing/28` | 28 px | | `height/h-32` | `spacing/32` | 32 px | | `height/h-36` | `spacing/36` | 36 px | | `height/h-40` | `spacing/40` | 40 px | | `height/h-44` | `spacing/44` | 44 px | | `height/h-48` | `spacing/48` | 48 px | #### Min-height Applied when a min height is set to a component or container, e.g., Textarea. | Utility variable | Aliased token | Value | | :- | :- | :- | | `min-height/min-h-4` | `spacing/4` | 4 px | | `min-height/min-h-8` | `spacing/8` | 8 px | | `min-height/min-h-16` | `spacing/16` | 16 px | | `min-height/min-h-20` | `spacing/20` | 20 px | | `min-height/min-h-24` | `spacing/24` | 24 px | | `min-height/min-h-28` | `spacing/28` | 28 px | | `min-height/min-h-32` | `spacing/32` | 32 px | | `min-height/min-h-36` | `spacing/36` | 36 px | | `min-height/min-h-40` | `spacing/40` | 40 px | | `min-height/min-h-44` | `spacing/44` | 44 px | | `min-height/min-h-48` | `spacing/48` | 48 px | #### Max-height Applied when a max height is set to a component or container, e.g., Textarea. | Utility variable | Aliased token | Value | | :- | :- | :- | | `max-height/max-h-4` | `spacing/4` | 4 px | | `max-height/max-h-8` | `spacing/8` | 8 px | | `max-height/max-h-16` | `spacing/16` | 16 px | | `max-height/max-h-20` | `spacing/20` | 20 px | | `max-height/max-h-24` | `spacing/24` | 24 px | | `max-height/max-h-28` | `spacing/28` | 28 px | | `max-height/max-h-32` | `spacing/32` | 32 px | | `max-height/max-h-36` | `spacing/36` | 36 px | | `max-height/max-h-40` | `spacing/40` | 40 px | | `max-height/max-h-44` | `spacing/44` | 44 px | | `max-height/max-h-48` | `spacing/48` | 48 px | | `max-height/max-h-52` | `spacing/52` | 52 px | | `max-height/max-h-56` | `spacing/56` | 56 px | | `max-height/max-h-60` | `spacing/60` | 60 px | | `max-height/max-h-64` | `spacing/64` | 64 px | | `max-height/max-h-68` | `spacing/68` | 68 px | | `max-height/max-h-72` | `spacing/72` | 72 px | | `max-height/max-h-76` | `spacing/76` | 76 px | | `max-height/max-h-80` | `spacing/80` | 80 px | | `max-height/max-h-84` | `spacing/84` | 84 px | | `max-height/max-h-88` | `spacing/88` | 88 px | | `max-height/max-h-92` | `spacing/92` | 92 px | | `max-height/max-h-96` | `spacing/96` | 96 px | | `max-height/max-h-100` | `spacing/100` | 100 px | ## Radius Corner radius (or radii) are the rounded corners in components. ### Guidelines * Radius size is dependent on the size, or height, of the component, with the exception of full rounded radii. * When applying a radius to non-Aura components, consider the size and scale. Use a larger radius for larger components and a smaller radius for smaller components. * Don't mix radius styles (e.g., rounded and sharp within the same component or UI element) as it can create inconsistency and visual clutter. ### Tokens | Utility variable | Aliased token | Value | | :- | :- | :- | | `border/rounded-none` | `radius/none` | 0 px | | `border/rounded-xs` | `radius/xs` | 2 px | | `border/rounded-sm` | `radius/sm` | 4 px | | `border/rounded-md` | `radius/md` | 6 px | | `border/rounded-lg` | `radius/lg` | 8 px | | `border/rounded-xl` | `radius/xl` | 12 px | | `border/rounded-2xl` | `radius/2xl` | 16 px | | `border/rounded-3xl` | `radius/3xl` | 24 px | | `border/rounded-4xl` | `radius/4xl` | 32 px | | `border/rounded-full` | `radius/full` | 9999 px | ## Stroke width Stroke thickness (or border weight) is used sparingly across the system to maintain a clean, minimal visual style. When applied, stroke weight should reinforce hierarchy, clarity, and accessibility without introducing visual noise. ### Guidelines * Avoid strokes unless necessary for clarity, separation, or emphasis. * When a stroke is required, use 1px for subtle borders or visual definition. * Ensure the appropriate border token color is used when a stroke is applied to an element. This ensures enough color contrast. * Use elevation and spacing first to define hierarchy—stroke should be a secondary tool. ### Tokens | Utility variable | Aliased token | Value | | :- | :- | :- | | `border-width/border-0` | `border-width/0` | 0 px | | `border-width/border-1` | `border-width/1` | 1 px | | `border-width/border-2` | `border-width/2` | 2 px | # Typography Source: https://docs.cognite.com/cdf/aura/foundations/typography Aura typefaces, type scale, and how typography tokens map to Inter and Source Code Pro. Typography is foundational to creating clear, consistent, and accessible interfaces. Our system uses design tokens and a well-defined set of styles to ensure type is applied consistently across products, platforms, and teams. Typography tokens are predefined values that represent font properties like font family, size, weight, letter spacing and line height. Rather than using arbitrary values (e.g., setting font-size: 17px), tokens standardize how typography is applied across components and layouts. Using type tokens: * Ensures consistency and scalability across the design system * Makes it easier to implement updates or refinements system-wide * Supports design–developer alignment through shared, meaningful naming conventions ## Typefaces | Token | Font | Use | | :- | :- | :- | | `--font-sans` / `--font-inter` | Inter | Main product copy — UI text, labels, and dense interfaces | | `--font-mono` | Source Code Pro | Code blocks, technical strings, terminal output | ## Type scale Styles are grouped by role: **Text** for product UI copy and **Code** for monospace technical content. ### Text Inter — main product typography. | Token | Description | Size | Line height | Weight | | :- | :- | :- | :- | :- | | `xxs/normal` | Metadata; short labels | 10px | 12px | Regular | | `xxs/medium` | Metadata; short labels; emphasized | 10px | 12px | Medium | | `xs/normal` | Small captions & chart labels | 12px | 14px | Regular | | `xs/medium` | Small captions & chart labels; emphasized | 12px | 14px | Medium | | `sm/normal` | Secondary text, small labels | 14px | 18px | Regular | | `sm/medium` | Secondary text, small labels; emphasized | 14px | 18px | Medium | | `base/normal` | Default body text | 16px | 20px | Regular | | `base/medium` | Default body text; emphasized | 16px | 20px | Medium | | `lg/normal` | Large text, sub-headlines | 18px | 24px | Regular | | `lg/medium` | Large text, sub-headlines; emphasized | 18px | 24px | Medium | | `xl/normal` | Headlines, small titles | 20px | 24px | Regular | | `xl/medium` | Headlines, small titles; emphasized | 20px | 24px | Medium | | `2xl/normal` | Page headers, titles | 24px | 28px | Regular | | `2xl/medium` | Page headers, titles; emphasized | 24px | 28px | Medium | | `3xl/normal` | Splash screens, onboarding or login | 30px | 32px | Regular | | `3xl/medium` | Splash screens, onboarding or login; emphasized | 30px | 32px | Medium | | `4xl/normal` | Splash screens, onboarding or login | 36px | 40px | Regular | | `4xl/medium` | Splash screens, onboarding or login; emphasized | 36px | 40px | Medium | | `5xl/normal` | Splash screens, onboarding or login | 48px | 44px | Regular | | `5xl/medium` | Splash screens, onboarding or login; emphasized | 48px | 44px | Medium | ### Code Source Code Pro — code blocks, snippets, and technical strings. | Token | Description | Size | Line height | Weight | | :- | :- | :- | :- | :- | | `sm` | Code blocks, snippets, and technical strings | 14px | 18px | Regular | | `md` | Code blocks, snippets, and technical strings | 16px | 20px | Regular | | `lg` | Code blocks, snippets, and technical strings | 18px | 24px | Regular | # Installing Aura Source: https://docs.cognite.com/cdf/aura/guides/installation Add Aura to a project, including the default setup that ships with Flows custom apps. Aura is still in active development. Expect frequent updates.

    An In progress badge means the topic is prioritized for development and sequencing. v1.0.0-style badges mark what is already stabilized.

    Aura is installed automatically with Flows, but you can also use it independently in your project. Add Aura to your project with the following command: `npm add @cognite/aura` Add the Aura skills to your project with the following command: `npx @cognite/cli apps skills pull --skill design` Update Aura to the latest version with the following command: `npm update @cognite/aura` ## Learn about Aura Foundations are the atomic visual styles and principles behind Aura. Open the overview to choose color, type, spacing, motion, and accessibility topics. Reusable UI building blocks. The index links to usage guidance and alphabetical reference pages. In progress \ Patterns are the higher-level building blocks that are composed of primitives to support complex workflows. In progress \ Contribute to the Aura design system and get help from the community. # Building with the Aura design system Source: https://docs.cognite.com/cdf/aura/index Explore Aura foundations, primitives, and developer guidance for building consistent, accessible product UI in Cognite Data Fusion (CDF) and Flows custom apps. Use these articles to understand Aura, adopt it in your projects, and build consistent product UI with shared foundations and primitives. Learn how Aura's AI-native foundations, primitives, and tooling support Cognite Data Fusion (CDF) and Flows custom apps. Understand color, typography, spacing, effects, interaction states, heuristics, and accessibility tokens. Browse Aura primitives by category, from data display and inputs to navigation and overlays. Extend or override Aura style tokens in a consuming app when scoped overrides are appropriate. See what the Aura team is building now and what is planned for upcoming releases. # About Aura Source: https://docs.cognite.com/cdf/aura/what-is-aura Learn about Aura, Cognite's AI-native design system. It combines a visual language, component library, and usage conventions teams and agents use to build consistent product UI in Cognite Data Fusion (CDF) and Flows custom apps. ## The AI-native differentiator Aura is a design system built for both human consumers and AI agents. Beyond components and guidelines, it includes the infrastructure, tooling, and structured documentation that enable AI-assisted development while ensuring implementations remain consistent with our standards. ### What "AI-native" means in practice A traditional design system ships components and documentation in isolated environments. Aura ships components, documentation, and a layer of structured AI tooling that makes correct implementation the path of least resistance for agents and humans all within the same context. * **Skills** — Structured prompt modules for Cursor, Claude, and other AI tools that activate automatically when a relevant pattern is detected. Skills enforce the right component, the right token, and the right pattern before any code is written. * **DESIGN.md** — A machine-readable source of truth for foundations and primitives. Agents can read this file to understand the system's constraints and intent, just as engineers would read documentation. * **LLM metadata** — Every component carries structured `llmUsage` metadata: when to use the component, when to reach for another primitive, and what pairs well with it. This metadata surfaces directly in agent context windows. * **MCP server** — A Model Context Protocol server that exposes Aura's documentation to AI tools at query time. Agents can retrieve up-to-date guidance without hallucinating from stale training data. ### Why this matters When an agent implements UI without design system guidance, it improvises. Improvisation accumulates into visual debt, broken theming, inaccessible patterns, and inconsistent experiences that are expensive to fix. Aura gives agents the same opinionated guardrails an engineer would enforce in code review, but earlier in the loop and at scale. ## How Aura differs from a traditional design system | Category | Traditional design system | Aura (AI-native) | | :- | :- | :- | | **What it ships** | Components and written documentation | Components, documentation, and structured AI tooling | | **Who consumes it** | Humans reading docs and Storybook | Humans and AI agents equally | | **Where guidance lives** | Notion, Confluence, or static sites | Machine-readable files (DESIGN.md, `llmUsage`, skills) | | **How tokens are applied** | Interpreted and manually applied by developers | Enforced at the skill layer before code is written | | **How conformance works** | Depends on human discipline in code review | Built into the agent workflow, not only code review | | **Agent tooling** | Out of scope | Skills, MCP server, and Cursor rules are first-class deliverables | The underlying visual language — tokens, components, patterns — is the same. The difference is in how that language is packaged and enforced. Aura treats agent-readable guidance as a core output of the design system, not an afterthought. ## What Aura defines Aura provides three layers that work together: | Layer | What it covers | Where it lives | | :- | :- | :- | | **Foundations** | Color, typography, spacing, radii, effects, motion, icons | Tokens, Figma, `DESIGN.md` | | **Primitives** | Component behavior, variants, composition rules, accessibility | NPM package, Storybook, `llmUsage` | | **Patterns & layouts** | Repeatable multi-component solutions: layouts, flows, surfaces | Skills, docs, Storybook | Aura defines the system constraints that product UI must operate within. It does not prescribe product-specific workflows — those are the responsibility of the teams building them. Aura ships in two tiers. **Core** components are production-ready, risk-reviewed, and available in the `@cognite/aura` package. **Labs** components are available for use but are still being stabilized and may change before graduating to Core. **Patterns** are in progress. See [In development](/cdf/aura/developer-guide/in-development) for the current roadmap. ## Who this documentation is for Aura serves three distinct audiences, and this documentation is organized with all three in mind. You work in the Figma library, prototype with Cursor or Claude, contribute to the system, and make decisions about how components look and behave. Use this documentation to understand usage intent, token semantics, and the design rationale behind system decisions. The Figma library is the canonical visual reference; this documentation explains the *why* behind it. **Start here:** [Color](/cdf/aura/foundations/color) → [Primitives overview](/cdf/aura/primitives) You implement product UI using the `@cognite/aura` package, with Storybook as your API reference and tokens as your styling system. Use this documentation for the decision-level guidance that doesn't live in props tables: when to reach for one component over another, how tokens are structured, and how to stay conformant without reading every Figma file. **Start here:** [Installing Aura](/cdf/aura/guides/installation) → [Storybook](https://cog.link/aura-storybook) You implement UI on behalf of engineers and designers. Consume Aura through its structured tooling layer: skills activate automatically for common patterns, `DESIGN.md` and `llmUsage` metadata provide scoped guidance, and the MCP server gives you access to up-to-date documentation at query time. Never invent component behavior or use raw hex values — always defer to the token and component system. **Start here:** [Installing Aura](/cdf/aura/guides/installation) → [Exploring Aura foundations](/cdf/aura/foundations) ## Related resources * [Building with the Aura design system](/cdf/aura) # Build solutions Source: https://docs.cognite.com/cdf/build/index Learn how to build and deploy your Cognite Data Fusion (CDF) solutions. Add the Industrial MCP server so an external AI tool can query CDF. Build, configure, and deploy AI agents and tools for advanced analytics. Compare Streamlit and Flows to determine the best path for your custom application needs. Visualize and monitor your data with Grafana, Power BI, and OData dashboards. Automate tasks and extend CDF with custom serverless functions. Analyze and visualize your data interactively using Jupyter notebooks in CDF. Build and deploy interactive data applications with Streamlit in CDF. Build and deploy production-ready React applications with scaffolding, authentication, and CDF integration. Use shared foundations, primitives, and patterns to build consistent, accessible product UI in Flows custom apps. Develop, train, and deploy machine learning models and data science workflows. # About the Flows custom app lifecycle Source: https://docs.cognite.com/cdf/flows/concepts/app-lifecycle Learn about draft and published versions, signing states, how to roll back, and which version Custom apps opens in Cognite Data Fusion (CDF). Every Flows custom app **version** has two independent states. **Lifecycle** says whether you can still change the files, either draft or published. **Signing state** says who has signed those files. CDF computes signing state from the signatures stored on the version. **Custom apps** opens one published version at a time. `apps activate` points **Custom apps** at a version, usually the newest, or an earlier Cognite-certified one if you need to roll back. ## How a version reaches Custom apps 1. **Deploy** — Upload a draft with [Deploying a Flows custom app](/cdf/flows/guides/deploying). You can still change the files. The version is **unsigned**. 2. **Builder signs** — A certified builder signs the bundle. The version is **builder-signed**. You can share a direct link. CDF shows a 14-day warning until Cognite certifies. See [Builder certification](/cdf/flows/guides/builder-certification) and [Code signing](/cdf/flows/guides/code-signing). 3. **Cognite certifies** — Submit the app for [Application certification](/cdf/flows/guides/application-certification). Cognite returns `.cert.sig` for the same bundle bytes. The version is **Cognite-certified** and still a draft. 4. **Publish** — Publishing locks the files. In production, you can publish only a Cognite-certified version. **Custom apps** does not change yet. 5. **Activate** — `apps activate` points **Custom apps** at that published version. You can point at the newest one, or an earlier Cognite-certified version to roll back. Every publish to production needs a **developer signature** and an **app certification signature** (`.cert.sig`). ## Version states Lifecycle is draft or published. Each row is a signing state, and each column is a lifecycle state. | Signing state | Draft | Published | | - | - | - | | **Unsigned** | The version owner can open it. Don't share yet. | Cannot publish. | | **Builder-signed** | Shareable by direct link. 14-day warning until Cognite certifies. | Cannot publish until Cognite-certified. | | **Cognite-certified** | Ready to publish. | Files cannot be overwritten. The active version is what **Custom apps** opens. | You cannot publish unsigned or builder-signed apps to production. ## New versions and rollback A bug fix, feature, or dependency bump is a **new version**. Testers use the builder-signed draft link, and **Custom apps** keeps the last Cognite-certified version until you activate the new one. Go through [Application certification](/cdf/flows/guides/application-certification) again for the new version. To roll back, point **Custom apps** at an earlier published, Cognite-certified version. You don't need to recertify. Do not run `apps deploy` against that `versionTag`. See [Roll back to a published version](/cdf/flows/guides/deploying#roll-back-to-a-published-version). ## Dev status Cognite can grant **Dev status** to some non-production projects. Those projects accept the developer signature alone, which is useful for development, staging, and sharing prototypes with end users. A builder-signed version runs there without the 14-day production warning. An unsigned version stays owner-only. Dev status is an exception for non-production work, not the standard way to publish. Use it only when you need to share a prototype or a staging build without [Application certification](/cdf/flows/guides/application-certification). Production projects always need both a developer signature and an app certification signature. See [Requesting Dev status for a CDF project](/cdf/flows/guides/dev-status). ## Further reading * [Deploying Flows custom apps](/cdf/flows/guides/deploying) — Interactive and CI/CD deployment, lifecycle commands, and rollback. * [Builder certification](/cdf/flows/guides/builder-certification) — Qualify to build Flows custom apps in customer environments. * [Code signing](/cdf/flows/guides/code-signing) — Sign bundles with a developer signature. * [Application certification](/cdf/flows/guides/application-certification) — Required for every app published to a production project. * [Dev status](/cdf/flows/guides/dev-status) — Skip application certification on non-production projects. # Flows custom app architecture patterns reference Source: https://docs.cognite.com/cdf/flows/concepts/architecture-best-practices Complete reference for architectural patterns and pitfalls in Flows custom apps. AI coding agents optimize for code that works now, not patterns that stay consistent. As Flows custom apps grow, implementations drift and become harder to review, test, and change safely. [Spec-driven development](/cdf/flows/concepts/spec-driven-development) helps keep requirements and implementation aligned when working with agents. Apply consistent patterns in these four areas to keep your codebase legible as your Flows custom app grows: * **Server state** → [TanStack React Query](https://tanstack.com/query/latest). * **Validated types** → [Zod](https://zod.dev) at the SDK boundary. * **Shared client state** → [Zustand](https://zustand.docs.pmnd.rs) (when state changes often) or React Context (for stable globals). * **Component shape** → business logic in hooks, components for rendering UI. This page is a pattern catalog with side-by-side Do/Don't examples. Read [Across all patterns](#across-all-patterns) before you simplify boilerplate. For a scan-friendly summary after the patterns, see [Quick reference](#quick-reference). ## Across all patterns These apply regardless of which patterns you follow. * **Input validation at trust boundaries.** Parse data from the SDK and from forms with Zod. TypeScript types are erased at runtime, so "the types already match" is not a reason to skip this. * **Error handling that prevents data loss.** Mutations should have `onError` rollbacks. A silent failure that leaves CDF in a partial state is harder to recover from than a visible error. * **CDF SDK only.** All CDF calls via `useCogniteSdk()`. Raw `fetch` or `axios` calls bypass CDF authentication controls and lose automatic token refresh, rate-limit retries, and request tracing. * **Security.** No secrets in client code, no `dangerouslySetInnerHTML` with unescaped user content. * **Accessibility.** Meaningful `aria` labels on interactive elements, keyboard navigability, no `onClick` on non-interactive elements like `
    `. * **Anything explicitly requested.** If a requirement, ticket, or review comment calls something out, it is not optional. For certification scoring on design and accessibility, see [Flows custom apps quality guidelines](/cdf/flows/guides/quality-guidelines). ## State: choosing the right tool Use the table below to pick the right state tool. It prevents most over- and under-engineering before you read the patterns. | State type | Tool | | - | - | | Local to one component | `useState` | | Stable global value (the CDF client, current user, feature flags) | React Context | | Frequently-updating shared state (selections, filters, wizard drafts) | Zustand | | Anything fetched from CDF | React Query — never copy into `useState` | React Context is the right default for stable globals. The Flows custom app scaffold already provides the CDF client via `CogniteSdkProvider`, so you thin-wrap its hook. Reach for Zustand when state changes frequently across many components or causes re-render storms. To reset all state in a component when an entity changes (for example, switching between contacts in an edit form), pass the entity's ID as `key`. React treats components with different keys as separate instances and resets their state automatically. No `useEffect` required. ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // No useEffect needed to reset form state when selectedContact changes ``` ## Recommended folder shape Group code by feature, not by technical type. Instead of top-level `hooks/`, `components/`, and `services/` folders that mix unrelated concerns, each feature gets its own folder with everything it needs: data, state, schema, and UI. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} src/ features/ alert-triage/ ui/ # presentational components specific to alert-triage context/ # context providers specific to alert-triage hooks/ # hooks specific to alert-triage useTriageVM.ts # view-model hook (owns data + commands) useAlerts.ts # query hook for this feature store.ts # Zustand: client state for alert-triage schema.ts # Zod validation schemas for alert-triage index.ts # public API of the feature context/ # Shared context providers hooks/ # Shared hooks stores/ # Shared Zustand store ui/ # Shared UI components ErrorState.tsx # one shared error surface App.tsx # shell + ErrorBoundary + QueryClientProvider ``` ## Patterns This section covers areas where Flows custom apps can drift as they grow. Where a pattern has a clear counterpart to avoid, a side-by-side example shows both. ### Use React Query for all server state A common pattern in growing Flows custom apps is fetching data with `useState` + `useEffect`. Developers often add a manual loading flag, a cancelled ref to handle race conditions, and a counter to force refetches. This hand-rolled loop is roughly what React Query does — except without caching, deduplication, background refresh, or retries. React Query ships with every Flows custom app; use it for all CDF reads and writes. See [TanStack Query in Flows](/cdf/flows/concepts/features#data-fetching-with-tanstack-query) for how the scaffold wires data fetching to CDF. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function useWellList(startDate: string, endDate: string) { return useQuery({ queryKey: ["wells", startDate, endDate], queryFn: async () => { const [wells, deferments] = await Promise.all([ fetchWells(), fetchDeferments(), ]); return joinWellData(wells, deferments); }, staleTime: 60_000, }); // caching, dedup, retries, background refetch: included } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function useWellList(startDate: string, endDate: string) { const [rows, setRows] = useState([]); const [isLoading, setIsLoading] = useState(true); const [error, setError] = useState(null); const [fetchKey, setFetchKey] = useState(0); // refetch hack useEffect(() => { let cancelled = false; setIsLoading(true); setError(null); Promise.all([fetchWells(), fetchDeferments()]) .then(([w, d]) => { if (!cancelled) setRows(joinWellData(w, d)); }) .catch((e) => { if (!cancelled) setError(e); }) .finally(() => { if (!cancelled) setIsLoading(false); }); return () => { cancelled = true; }; }, [fetchKey, startDate, endDate]); const refetch = () => setFetchKey((k) => k + 1); // no cache, no dedup, no bg refresh, no retry } ``` Query keys must be unique and specific to avoid cache collisions — a collision silently serves wrong data and is difficult to debug in production. Always include the resource type and all relevant parameters. Keys should mirror the shape of the underlying API request: a `GET /projects?userId=123&status=active` call maps to `["projects", userId, status]`. `useState` is intended for local UI state (modals, form inputs, toggles). React Query handles server state and API data — never copy query data into `useState`, which opts out of background updates and causes the UI cache to drift from the server. ### Parse at the SDK boundary with Zod CDF instance properties are loosely typed by nature. Without an explicit parse step, that looseness leaks into the rest of your Flows custom app: every component that needs a value ends up re-asserting the same assumptions with `as` casts and ad-hoc coercers. Parse the response into a known shape once, at the point where it enters your Flows custom app, and downstream code can trust the type without re-asserting it. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // schema.ts — one schema per view; z.infer is the type const WellSchema = z.object({ name: z.string().optional(), uwi: z.string().nullish(), latitudeWGS84: z.coerce.number().default(0), longitudeWGS84: z.coerce.number().default(0), }); type Well = z.infer; function mapNode(node: NodeDefinition): Well { const raw = node.properties?.[VIEW.space]?.[KEY] ?? {}; return WellSchema.parse(raw); // wrong shape → throws here, surfaced as a query error // not silently wrong data } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function getViewProps(node: NodeDefinition) { const bySpace = (node.properties as Record>) ?.[VIEW.space]; return (bySpace?.[KEY] as Record) ?? {}; } function mapNode(node: NodeDefinition): Well { const p = getViewProps(node); return { name: (p.name as string) ?? node.externalId, uwi: p.uwi as string, latitudeWGS84: Number(p.latitudeWGS84) || 0, // if latitudeWGS84 is renamed upstream, this silently // becomes 0 — no error, ever }; } ``` Parse at exactly two seams: SDK → app (responses) and form → SDK (writes). Downstream functions receive a real typed value, not `unknown`. ### Keep data access thin It's tempting to wrap CDF calls in service classes with interfaces for testability, but for a read-mostly UI, the overhead rarely pays off. An interface, a class, a ServicesContext, and a per-hook DepsContext to perform a single `list()` means touching four files for what should be one hook. A typed query hook is sufficient. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // useAlerts.ts — one obvious place per resource export function useAlerts() { const client = useCogniteSdk(); return useQuery({ queryKey: ["alerts"], queryFn: async () => { const res = await client.instances.list(ALERT_SOURCE); return res.items.map((node) => { const raw = node.properties?.[VIEW.space]?.[KEY] ?? {}; return AlertSchema.parse(raw); // consumers receive Alert[] — never unknown }); }, }); } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // services/cpv-plan-service.ts export interface CpvPlanService { listPlans(): Promise; } export class ApiCpvPlanService implements CpvPlanService { /* ... */ } // contexts/services-context.tsx const defaultServices = { cpvPlanService: new MockCpvPlanService(), // mock as the default! }; export const useServices = () => useContext(ServicesContext); // + a per-hook DepsContext just to inject a mock in tests ``` A service class earns its place when there's real logic — pagination, joining, write-shaping — or a genuine need to swap implementations. Start with a typed query hook; promote only when the body actually grows. All CDF API calls must go through the SDK via `useCogniteSdk()`. Never make raw `fetch` or `axios` requests to CDF endpoint URLs. The SDK handles token refresh, retries on rate-limit errors (429s), typed error responses, and consistent request tracing — none of which you get from a plain fetch. ### Choose the right home for client state React Context and Zustand solve different problems. Context re-renders every consumer when its value changes, which suits values that rarely change. Zustand re-renders only components that subscribe to the changed slice, which suits values that change frequently. Use Context for app-shell configuration that is set once and read everywhere, for example, the CDF client, current user, and feature flags. Use Zustand for state that updates during user interaction, such as selections, filters, and wizard drafts, or that many unrelated components read. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // store.ts — colocated with the feature // Use Zustand for state that changes often (filters, selection) export const useTriageStore = create((set) => ({ selectedId: null, filters: defaultFilters, select: (id) => set({ selectedId: id }), setFilters: (f) => set((s) => ({ filters: { ...s.filters, ...f } })), })); // Components subscribe only to the slice they need function Row({ id }: { id: string }) { const selected = useTriageStore((s) => s.selectedId === id); const select = useTriageStore((s) => s.select); // re-renders only when this row's selected state changes } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // State threaded through every level as props // Or a hand-rolled Context per concern const Ctx = createContext(null); function Provider({ children }) { const [sel, setSel] = useState(null); return {children}; } // Every consumer re-renders when any value in the context changes ``` Start with `useState` for local state. Lift to Context when a stable value needs to be available across the tree. Reach for Zustand when that value starts changing frequently, when you're prop-drilling beyond 3 levels, or when Context re-renders become noticeable. ### Separate data from display When components grow past a few hundred lines, they are usually doing too much: fetching their own data, computing derived values, and rendering in one place. That makes them harder to test in isolation and easier to break with unrelated edits. Extract data and logic into a view-model hook instead, and let the component focus on rendering. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // hook owns data & commands — testable in isolation function usePipelineRun() { const run = useMutation({ mutationFn: runPipeline }); return { run: run.mutate, isPending: run.isPending, error: run.error }; } // component is a thin binding to the design system function PipelinePanel() { const { run, isPending, error } = usePipelineRun(); return ( {error && {error.message}} ); } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function PipelinePanel({ sdk }: { sdk: CogniteClient }) { const [loading, setLoading] = useState(false); const [error, setError] = useState(null); // dozens more useState, inline service calls, business logic… return (
    {error &&
    {error}
    }
    ); } ```
    ### No business logic in UI components This extends the view-model pattern above. When agents or developers add features quickly, business logic accumulates in components: a `useQuery` call here, a `useEffect` fetch there, an inline command handler that should live in a hook. Over time the component becomes the source of truth for data fetching and commands that belong in hooks. Keep components responsible for rendering and local UI state only; everything else belongs in a hook instead. ```tsx Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function FormUI() { // business logic lives in the hook const { data, isLoading, addData } = useFormBusinessLogic(); // UI state stays in the component const [modalOpen, setModalOpen] = useState(false); // simple UI events are fine const handleChange = (e: ChangeEvent) => { // ... }; return
    ...
    ; } ``` ```tsx Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function FormUI() { // BAD: data fetching in the component const { data } = useQuery({ queryKey: ["form"], queryFn: fetchFormData }); const { mutate } = useMutation({ mutationFn: submitForm }); // BAD: business command defined in the component body useEffect(() => { api.fetch().then(setState); }, []); const addData = useCallback(() => { api.addData(); }, []); return
    ...
    ; } ```
    Modal open/close, hover, and resize observers are UI state. Keep them in the component. ### Compute derived values during rendering When a value can be computed from existing props or state, the temptation is to store a copy in `useState` and keep it in sync with `useEffect`. This is unnecessary: React re-runs the component on every relevant state change anyway, and an inline `const` is always up to date. The `useEffect` version causes two renders: one with the stale value, one after the Effect fires, and adds a copy that can drift from its source. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Always up to date — no Effect needed const filtered = items.filter((i) => i.active); // For genuinely expensive transforms, memoize const sorted = useMemo(() => items.slice().sort(byDate), [items]); ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Extra state that can drift from its source const [filtered, setFiltered] = useState([]); useEffect(() => { setFiltered(items.filter((i) => i.active)); }, [items]); // Causes two renders: one with the stale value, one after the Effect fires ``` The [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect) React article covers the full range of cases where `useEffect` is the wrong tool: derived state, event handlers, parent notification, and more. ### Keep hooks single-purpose As features grow, hooks often return unrelated values together: query data, derived fields, and callbacks in one object. A component that uses only one of those values still re-renders when any other value changes. Split hooks by responsibility, not by feature area. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Split into focused hooks — each component subscribes to what it needs const useUserStats = () => { // ... return { userStats }; }; const useChartData = () => { // ... return { chartData }; }; function UserStatsComponent() { const { userStats } = useUserStats(); // only re-renders when userStats changes return
    {userStats.total}
    ; } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} const useDashboardLogic = () => { // lots of unrelated logic return { userStats, chartData, recentActivity, settings }; }; function UserStatsComponent() { // re-renders when chartData or recentActivity change, // even though this component only uses userStats const { userStats } = useDashboardLogic(); return
    {userStats.total}
    ; } ```
    Signal to split: multiple components call the same hook but destructure completely different values. ### Expose a narrow interface from hooks Returning the full `query` or `mutation` object from a hook is tempting, but it leaks React Query internals to every caller. If you later change how data is fetched, such as swapping a query for a subscription or adding a transformation, you must update every component that destructures the raw object. Expose only what callers need: data, a loading flag, and named command functions. ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function useFormBusinessLogic() { const query = useQuery({ queryKey: ["form"], queryFn: fetchFormData }); const mutation = useMutation({ mutationFn: submitForm }); // wrap mutations in named command functions const addData = (data: DataType) => mutation.mutate(data); // derive cheap values inline const itemCount = query.data?.length ?? 0; // memoize only when profiling shows a real cost const sortedItems = useMemo( () => query.data?.slice().sort(byDate), [query.data] ); return { data: query.data, isLoading: query.isLoading, addData, itemCount, sortedItems, }; } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function useFormBusinessLogic() { const query = useQuery({ queryKey: ["form"], queryFn: fetchFormData }); const mutation = useMutation({ mutationFn: submitForm }); const store = useMyStore(); // raw store exposed directly // returning full objects leaks internals to every caller return { query, mutation, store }; } ``` ### Make impure dependencies injectable Hooks that call `useQuery`, `useMutation`, routing hooks, or SDK clients are impure. Hard-coded imports make them difficult to test without `vi.mock`. That workaround is path-coupled and not type-checked. It can also break silently when you rename or move a file. There are two ways to make dependencies injectable. Choose based on the unit's depth in the tree, and be consistent within a unit: * For shallow units, add a `deps` prop with real implementations as the default. Tests pass a typed substitute directly, no mocking needed. * For containers and nested trees, put dependencies in a `.context.tsx` file with real defaults so production code needs no Provider. Tests inject via ``. ```tsx Do — Default-Props DI wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // tasks.tsx import { useTasks as useTasksImpl } from "../services/use-tasks"; import type { Task } from "../types"; type TasksDeps = { useTasks: () => { isLoading: boolean; tasks: Task[] }; }; const defaultDeps: TasksDeps = { useTasks: useTasksImpl }; export const Tasks = ({ deps = defaultDeps }: { deps?: TasksDeps }) => { const { isLoading, tasks } = deps.useTasks(); if (isLoading) return ; return
      {tasks.map((t) => )}
    ; }; // tasks.spec.tsx — no vi.mock needed const deps: ComponentProps["deps"] = { useTasks: vi.fn(() => ({ isLoading: false, tasks: [{ id: "1", text: "a" }] })), }; render(); ``` ```tsx Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // Discouraged for first-party modules: path-coupled, not type-checked. // Renaming use-tasks.ts breaks this test without a type error. import { useTasks } from "../services/use-tasks"; vi.mock("../services/use-tasks", () => ({ useTasks: () => ({ isLoading: false, tasks: [] }), })); ```
    `vi.mock` remains appropriate for third-party and external libraries. For first-party code, prefer a typed seam. If you find yourself prop-drilling `deps` through intermediate components, switch to Context DI instead. ### Use Aura instead of inline styles Inline `style={{}}` objects bypass the `aura/no-overriding-styles` ESLint guardrail and guarantee visual drift from the [Aura design system](/cdf/aura). They also signal that the component is doing too much: they often appear in oversized components that also mix in data fetching and business logic. If you need to customize the appearance or styling, follow this order of priority: 1. Use the component props and variants available in the design system. 2. Use [Tailwind utility classes](https://tailwindcss.com/docs/styling-with-utility-classes) for additional styling. 3. If neither option is sufficient, raise a support ticket with the Aura team. For details on supported customization options, see the [Aura customization documentation](/cdf/aura/developer-guide/customization). ```tsx Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function StatusCard({ status }: { status: "ok" | "error" }) { return ( {status} ); } ``` ```tsx Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function StatusCard({ status }: { status: "ok" | "error" }) { return (
    {status}
    ); } ```
    The `aura/no-overriding-styles` ESLint rule enforces this automatically. Don't disable or suppress it. Browse [Aura primitives](/cdf/aura/primitives) for layout and feedback components — for example [Card](/cdf/aura/primitives/card) and [Badge](/cdf/aura/primitives/badge) — instead of styled HTML elements. ### Centralize your data model config Do not scatter view IDs, space names, and property keys as string literals across your codebase. When you port a Flows custom app to another CDF project, or rename a space or view, you must find and replace every occurrence by hand. Keep all identifiers in `config/model.ts` and import from there instead. For batching and concurrency when you read and write instances, use the `dm-limits-and-best-practices` builder skill ([About builder skills](/cdf/flows/concepts/skills)). ```typescript Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // config/model.ts — one place for all CDF view identifiers export const MODEL = { space: "my-solution-space", views: { well: { externalId: "Well", version: "1" }, }, properties: { well: { name: "name", uwi: "uwi", lat: "latitudeWGS84" }, }, } as const; // Changing the space name means editing one line here. // All usages across the codebase update automatically. function useWells() { const client = useCogniteSdk(); return useQuery({ queryKey: ["wells"], queryFn: () => client.instances.list({ sources: [{ source: { type: "view", ...MODEL.views.well, space: MODEL.space } }], }), }); } ``` ```typescript Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // useWells.ts function useWells() { const client = useCogniteSdk(); return useQuery({ queryKey: ["wells"], queryFn: () => client.instances.list({ sources: [{ source: { type: "view", externalId: "Well", version: "1", space: "my-solution-space" } }], }), }); } // useWellFilters.ts — same literals, different file const filter = { equals: { property: ["my-solution-space", "Well/1", "uwi"], value: uwi } }; // WellDetail.tsx — and again const lat = node.properties?.["my-solution-space"]?.["Well/1"]?.["latitudeWGS84"]; // Rename the space or bump the view version and you'll miss at least one. ``` ### Handle errors in two layers Without a defined error strategy, each component handles failures differently. Some store error text in state and render a red `
    `. Others fail silently with an empty screen. An uncaught render error can crash the whole iframe. Use two layers for most cases.
    
    
      ```tsx Do wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
      // App.tsx — one boundary at the shell catches render crashes
      }>
        
          
        
      
    
      // Per view — use the query's own error state
      const { data, error, isError, refetch } = usePlans();
      if (isError) return ;
    
      //  is one shared component used everywhere
      ```
    
      ```tsx Don't wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
      // Per-component string + red pre
      try {
        const data = await svc.load();
        setData(data);
      } catch (e) {
        setError(e instanceof Error ? e.message : String(e));
      }
    
      {error && 
    {error}
    } // A render crash takes down the whole iframe. // Retry = reload the page. ```
    ## Quick reference The following table summarizes the patterns on this page. | Stop doing | Start doing | | - | - | | Fetching with `useState` + `useEffect` + `fetchKey` | All CDF reads and writes behind `useQuery`/`useMutation` in a dedicated hook | | Chained `as` casts at every call site | One Zod schema per view; `parse()` at both seams; type via `z.infer` | | Interface + class + two context layers for a single `list()` | Typed `use*` hook with `useQuery` inline | | Prop-drilling shared state beyond 3 levels | Zustand for frequently-updating state; Context for stable globals | | Components that fetch, compute, and render | View-model hook + thin Aura-built component; `useQuery`/`useMutation` only in hooks, never in `.tsx` files | | Hooks that return full `query`/`mutation` objects | Narrow return: `{ data, isLoading, addData }` | | One hook returning everything for a domain | Focused single-purpose hooks per concern | | `vi.mock` on first-party modules to bypass missing seams | Typed DI (Default-Props or Context) | | Inline `style={{}}` objects | Aura/Tailwind primitives enforced by `aura/no-overriding-styles` | | View IDs and space names scattered as string literals | Central `config/model.ts` | | Per-component error strings / red `
    ` | One `` at the shell + shared `` |
    
    ## Related topics
    
    * [About Flows custom app features](/cdf/flows/concepts/features) — Scaffolding, TanStack Query, auth, and what ships with every Flows custom app.
    * [Auth API](/cdf/flows/reference/api/auth) — `connectToHostApp` and authenticated SDK access.
    * [Host App API](/cdf/flows/reference/api/host-app-api) — Navigation, state, sidebar visibility, fullscreen, and agent integration.
    * [About spec-driven development](/cdf/flows/concepts/spec-driven-development) — Checked-in specs for you and your agent.
    * [About builder skills](/cdf/flows/concepts/skills) — Skills including `design`, `dm-limits-and-best-practices`, and `security`.
    * [Flows custom apps quality guidelines](/cdf/flows/guides/quality-guidelines) — Certification rubric including Aura consistency.
    * [Aura design system](/cdf/aura) — Components, tokens, and patterns for Flows custom app UIs.
    * [Get started with Flows](/cdf/flows/guides/getting-started) — Create a project and run it in CDF.
    
    
    # About Flows custom app features
    Source: https://docs.cognite.com/cdf/flows/concepts/features
    
    Overview of Flows features for building custom apps for Cognite Data Fusion (CDF): scaffolding, authentication, Vite local development, TanStack Query, testing, code quality, TypeScript, and deployment.
    
    **Flows** (`@cognite/cli`) is a CLI for building production-ready React apps that run inside **CDF**. You get project scaffolding, authenticated access to the Cognite SDK, Vite-based local development, and paths to deploy without assembling those pieces yourself.
    
    For a hands-on path from install to first run, see [Get started with Flows](/cdf/flows/guides/getting-started).
    
    ## Scaffolding
    
    You create a full app layout, configuration, and recommended tooling with a single command:
    
    ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    npx @cognite/cli@latest apps create
    ```
    
    The generator captures org, project, and cluster in `app.json` and sets up **Cursor** rules when you use Cursor as your editor. See [Get started with Flows](/cdf/flows/guides/getting-started#create-and-run-your-app) for prompts and folder layout.
    
    ## UI with Aura
    
    Generated apps include **`@cognite/aura`**. Use the [Aura design system](/cdf/aura) for foundations, primitives, and patterns, and [Installing Aura](/cdf/aura/guides/installation) to add or upgrade the package.
    
    ## Authentication in CDF
    
    Flows integrates with CDF sign-in when your app runs in the CDF iframe. Call **`connectToHostApp()`** from **`@cognite/app-sdk`** to get a **`HostAppAPI`** instance, then use `getProject()`, `getBaseUrl()`, and `getAccessToken()` to construct an authenticated **`CogniteClient`**.
    
    See the [Flows Auth API](/cdf/flows/reference/api/auth) for props, loading and error, and return values.
    
    To start an additional Microsoft Entra ID sign-in from within a Flows custom app, see [Signing in with an additional Microsoft Entra ID account](/cdf/flows/guides/oauth-relay).
    
    ## Local development with Vite
    
    Templates use **Vite** with **hot module replacement (HMR)**, **HTTPS** on localhost (locally trusted certificates), and optional **auto-open in CDF** via `fusionOpenPlugin`. With auto-open, you run the dev build inside the CDF iframe and hit the same sign-in flow and URLs you get when the app is published.
    
    See [Run your app locally](/cdf/flows/guides/running-locally) and the [Vite plugin API](/cdf/flows/reference/api/vite).
    
    ## Build and deploy
    
    Deploy to **CDF** directly:
    
    ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    npm run deploy
    ```
    
    Preview deployments for pull requests:
    
    ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    npm run deploy-preview
    ```
    
    For interactive deploy and CI/CD, see [Deploy your Flows app](/cdf/flows/guides/deploying).
    
    ## Testing
    
    Generated custom apps include **Vitest** and **React Testing Library**. From your app directory, run:
    
    ```bash wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    npm test           # Run tests
    npm run test:watch # Watch mode
    npm run test:ui    # Visual UI
    ```
    
    Script names are defined in `package.json`.
    
    ## Code quality
    
    The template ships with **ESLint** configuration (`eslint.config.mjs`) and npm scripts for linting and formatting. From your app directory, run:
    
    ```bash wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    npm run lint       # Check for issues
    npm run lint:fix   # Auto-fix issues
    ```
    
    Script names are defined in `package.json`.
    
    ## Data fetching with TanStack Query
    
    **TanStack Query (React Query)** is included so you can pair cached, declarative data loading with an authenticated SDK from `connectToHostApp()`:
    
    ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
    import { useQuery } from "@tanstack/react-query";
    import { CogniteClient } from "@cognite/sdk";
    
    function Assets({ sdk }: { sdk: CogniteClient }) {
    
      const { data, isLoading } = useQuery({
        queryKey: ["assets"],
        queryFn: () =>
          sdk.instances.list({
            instanceType: "node",
            limit: 10,
            sources: [
              {
                source: {
                  type: "view",
                  space: "cdf_cdm",
                  externalId: "CogniteAsset",
                  version: "v1",
                },
              },
            ],
          }),
      });
    
      if (isLoading) return 
    Loading assets...
    ; return (
      {data?.items.map((asset) => (
    • {asset.properties?.["cdf_cdm"]?.["CogniteAsset/v1"]?.name}
    • ))}
    ); } ``` ## TypeScript Apps are **TypeScript-first**, with strict settings suited to app development. Types from **`@cognite/sdk`** (for example, resources such as assets) work with your components and hooks as you build out the UI. ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import type { Asset } from "@cognite/sdk"; function AssetCard({ asset }: { asset: Asset }) { return (

    {asset.name}

    {asset.description}

    ); } ``` ## Further reading * [Get started with Flows](/cdf/flows/guides/getting-started) — Create a project and run it in CDF. * [Aura design system](/cdf/aura) — Foundations, primitives, and patterns for Flows custom app UI. * [Run your app locally](/cdf/flows/guides/running-locally) — Dev server, HTTPS, and troubleshooting. * [Signing in with an additional Microsoft Entra ID account](/cdf/flows/guides/oauth-relay) — Additional Microsoft Entra ID sign-in via the App Hosting OAuth relay. * [Deploy your Flows app](/cdf/flows/guides/deploying) — Interactive deploy and CI/CD. # About Flows custom apps Source: https://docs.cognite.com/cdf/flows/concepts/index Understand the core concepts of Flows custom apps, from built-in features and AI skills to spec-driven development for Cognite Data Fusion (CDF) apps. Learn what **Flows** (`@cognite/cli`) gives you out of the box and how to align your team around specs before you dive into procedures and APIs. Scaffolding, auth, Vite, tests, deploy, and data fetching at a glance. Agent-agnostic AI skills that ship with every Flows custom app. Plan features in checked-in specs your team and agent both work from. Draft vs published versions and how the active version works. Patterns and pitfalls for keeping Flows apps well-structured as they grow. # About builder skills Source: https://docs.cognite.com/cdf/flows/concepts/skills Learn about builder skills, how they ship with every app, which AI development environments are supported, and how to keep skills up to date. **Skills** extend AI agent capabilities with specialized knowledge and workflows. The Flows builder skills help your AI coding agents build and deploy Flows custom apps for **Cognite Data Fusion (CDF)**. They capture Cognite best practices, API guidance, and opinionated workflows so agents such as Claude Code, Cursor, GitHub Copilot, Windsurf, and any agent that follows the [Agent Skills](https://agentskills.io/home) standard stay aligned from the first prompt. ## How skills are delivered When you run `npx @cognite/cli@latest apps create`, the CLI fetches skills automatically from the [`cognitedata/builder-skills`](https://github.com/cognitedata/builder-skills) repository and writes them into your app. No extra install or configuration — compatible agents read the skill files from your workspace. The generated app places skills in two directories so that compatible agents can locate them automatically: | Directory | Supported coding agents | | - | - | | `.agents/skills/` | Cursor, Windsurf, Cline, Continue.dev, Aider, OpenHands, Devin, and any agent following the [Agent Skills](https://agentskills.io/home) standard | | `.claude/skills/` | **Claude Code** (Anthropic) — dedicated path for Claude Code skill files | In addition, two root-level context files give agents project-wide instructions at the start of each session: | File | Supported coding agents | | - | - | | `AGENTS.md` | Cursor, GitHub Copilot, Windsurf, Cline, Aider, OpenHands, Devin, and most other agents | | `CLAUDE.md` | **Claude Code** — project memory file loaded at the start of every Claude Code session | ## Keep skills up to date Cognite updates the `builder-skills` repository regularly. To pull the latest skills into an existing app, run: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps skills pull ``` To pull a specific skill: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps skills pull --skill ``` Pull skills after upgrading `@cognite/cli` or when starting new work that touches authentication, data modeling, or Atlas agents — areas where Cognite updates skills most often. ## Example skills The following table lists skills that ship with new apps. | Skill | Purpose | | - | - | | `setup-flows-auth` | Wire up `connectToHostApp` and authenticated CDF access. | | `integrate-atlas-chat` | Add a streaming **Atlas Agent** chat interface to your app. | | `create-client-tool` | Generate an `AtlasTool` and integrate it with `useAtlasChat`. | | `design` | Apply **[Aura](/cdf/aura)** UI components, tokens, layouts, and accessibility guidance. | | `dm-limits-and-best-practices` | Follow **CDF** data modeling concurrency and batching guidance. | | `performance` | Optimize for speed, render counts, and bundle size. | | `security` | Review for credentials, user input, and external data vulnerabilities. | | `flows-app-brief` | Coach you through the App Brief before you build. | | `flows-code-review` | Run the technical review and surface Must Fix items. | | `flows-design-review` | Score the app against the 10 design quality questions. | | `flows-external-app-submit` | Verify all artifacts and submit for [Application certification](/cdf/flows/guides/application-certification). | For the full list and the source of each skill, see [`cognitedata/builder-skills`](https://github.com/cognitedata/builder-skills). ## Supported AI development environments Flows custom apps work with **any modern AI coding agent**. Every generated app includes builder skills that agents detect and use automatically without manual configuration. **Claude Code is fully supported.** Every generated app includes a dedicated `.claude/skills/` directory and a `CLAUDE.md` project context file, giving Claude Code immediate, structured access to Cognite development guidelines. The table below shows which environments are supported, where each agent reads skills from, and what project context file it uses. If your preferred environment is not listed, it can still work via `AGENTS.md`. | Agent / Environment | Vendor | Skills directory | Project context file | MCP support | Status | | - | - | - | - | - | - | | **Cursor** | Anysphere | `.agents/skills/` | `AGENTS.md` | Yes (`.cursor/mcp.json`) | Recommended | | **Claude Code** | Anthropic | `.claude/skills/` | `CLAUDE.md` | Yes | Fully supported | | **GitHub Copilot** | Microsoft / GitHub | `.agents/skills/` | `AGENTS.md` | Limited (VS Code only) | Supported | | **Windsurf** | Codeium | `.agents/skills/` | `AGENTS.md` | Yes | Supported | | **Google Gemini (Antigravity)** | Google | `.agents/skills/` | `AGENTS.md` | Planned | Supported | | **Amazon Q Developer** | Amazon | `.agents/skills/` | `AGENTS.md` | Limited | Supported | | **Continue.dev** | Continue | `.agents/skills/` | `AGENTS.md` | Yes | Supported | | **Cline** | Community / Anthropic | `.agents/skills/` | `AGENTS.md` | Yes | Supported | | **Aider** | Aider-chat | `.agents/skills/` | `AGENTS.md` | No | Supported | | **OpenHands** | All Hands AI | `.agents/skills/` | `AGENTS.md` | No | Supported | | **Devin** | Cognition AI | `.agents/skills/` | `AGENTS.md` | No | Supported | | **JetBrains AI Assistant** | JetBrains | `.agents/skills/` | `AGENTS.md` | No | Partial | | **Zed AI** | Zed Industries | `.agents/skills/` | `AGENTS.md` | Yes | Partial | | **VS Code (Copilot Chat)** | Microsoft | `.agents/skills/`. | `AGENTS.md` | Limited | Supported | | **Gemini CLI** | Google | `.agents/skills/` | `AGENTS.md` | Yes | Supported | | **Amp** | Sourcegraph | `.agents/skills/` | `AGENTS.md` | Yes | Supported | **Column legend:** * **Skills directory** — where the agent reads builder skill files from the generated app * **Project context file** — the root-level file the agent reads for project-wide instructions * **MCP support** — whether the environment supports Model Context Protocol tool servers, useful for connecting to CDF APIs directly from your agent Skills are included in both `.agents/skills/` (for Cursor and the [Agent Skills](https://agentskills.io/home) standard) and `.claude/skills/` (for Claude Code). To update skills, run `npx @cognite/cli@latest apps skills pull`. ## Further reading * [Get started with Flows](/cdf/flows/guides/getting-started) — Create an app and see skills land in your workspace. * [Flows features](/cdf/flows/concepts/features) — Everything the template gives you at a glance. # About spec-driven development Source: https://docs.cognite.com/cdf/flows/concepts/spec-driven-development Learn how spec-driven development works in Cognite Flows: a checked-in SPEC.md for each app, or optional spec-kit slash commands for structured per-feature planning before you write code. In Flows, you treat checked-in specs as the source of truth for a feature: they align your team and give your agent the same brief every time you return to the repo. ## Specs vs. plan mode **Plan mode** (for example in Claude Code, or similar in-session planning in other AI coding tools) fits **one conversation**: you think through the task now, and your agent can act on that plan immediately. **Repo-native specs** live in **git**, survive across sessions, can go through **review** like code, and travel with the feature from first sketch to release. You refine the spec as you learn, share it with collaborators, and reuse it as **stable context** whenever you open the project with your agent. The approaches stack. You can use plan mode **inside** a spec-kit flow — for example, reason in-session, then run `/speckit.tasks`. Specs anchor **what** you are building; in-session planning helps with **how** you execute the next slice. ## Two modes in Flows When you run `npx @cognite/cli@latest apps create`, the CLI asks how you want to plan your app. | Mode | When to use | | - | - | | **Lightweight `SPEC.md` (default)** | One structured `SPEC.md` at the repository root. No spec-kit bundle or slash commands. Solo work, small apps, early ideation, or a single spec for the whole app. | | **[Spec-kit](#spec-kit) (opt-in)** | Per-feature specs under `specs/-/` and slash commands in Claude Code and Cursor, using a curated, vendored subset of [spec-kit](https://github.com/github/spec-kit). Teams, multi-session work, or several distinct features. | The spec-kit prompt defaults to **No**. If you skip it and want spec-kit later, there is no retrofit command — you would need to re-scaffold or manually copy the [spec-kit files](#spec-kit) into your app. ## Lightweight SPEC.md By default, your Flows custom app includes `SPEC.md` at the repository root. Use it as a living product spec: the template captures user intent, requirements, and how the app integrates with CDF data models. The following table describes each section. | Section | Purpose | | - | - | | **User Stories** | Who uses the app, what they want, and why. | | **Acceptance Scenarios** | Given / When / Then statements. | | **Functional Requirements** | Numbered FRs (FR-001, FR-002, …) plans and tasks can reference. | | **Success Criteria** | Measurable, user-visible outcomes. | | **Clarifications** | Open questions before planning starts. | | **Assumptions** | Scoping decisions and out-of-scope items. | | **Data Models & CDF Integration** | Existing views, new views, and spaces *(mandatory)*. | **Example**: a populated excerpt for a **Production Accounting dashboard**. This is a CDF app that shows oil balance KPIs and a hierarchical reconciliation tree with quality-flagged flow components: ```markdown wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ## User Stories ### User Story 1 — View at-a-glance production KPIs (Priority: P1) A production accountant opens the dashboard to get an immediate snapshot of oil balance health. The KPI bar surfaces four headline metrics — Gross Production, Net Export, Imbalance, and Closure % — so the accountant can tell at a glance whether the period is closing within acceptable tolerance. ### User Story 2 — Explore the Balance Reconciliation Tree (Priority: P2) The accountant needs to trace the source and destination of production volumes. The Balance Reconciliation Tree is a hierarchical table of flow components (Wellhead Production → Field Gross → Terminal Receipts → Pipeline Export → Losses). Each row shows the component name, a numeric value with a unit, a source tag, and a quality badge (Verified / Estimated / Manual). ## Functional Requirements - FR-001: The dashboard MUST display a KPI bar with four metrics: Gross Production (Mbbl), Net Export (Mbbl), Imbalance (Mbbl), and Closure (%). Imbalance MUST be derived from tree data; Closure % MUST equal Net Export / Gross Production × 100. - FR-004: The Balance Reconciliation Tree MUST render as a hierarchical table with collapsible parent rows and child rows. - FR-006: Quality badges MUST support exactly three states: Verified (success tone), Estimated (warning tone), Manual (neutral tone). ## Edge Cases - A flow component with no source tag MUST render an em-dash (—) placeholder, never a blank cell. - When Imbalance is exactly zero, the Imbalance KPI MUST render in a neutral success tone (fully balanced). - When Closure % exceeds 100%, the value MUST render as-is in a warning tone (over-allocation). ## Success Criteria - SC-001: The KPI bar renders within 1 second of load. - SC-003: A user can locate any flow component in the tree in under 30 seconds using expand/collapse and the quality filter. - SC-005: The tree supports at least 3 levels of hierarchy without layout breakage. ## Assumptions - Each balance covers a single reporting period; a period picker is out of scope for v1. - Volumes are in Mbbl (thousand barrels); Closure is a percentage. - The app embeds in the Cognite Data Fusion iframe; auth is delegated to the host. Export and print are out of scope for v1. ## Data Models & CDF Integration ### Existing views - `cdf_cdm.CogniteTimeSeries:v1` — meter readings and production rates behind each source tag (for example, `FLD-GROSS-001`, `LIFT-TOT-001`). ### New views - `production_accounting.FlowComponent:v1` — one node per balance line. Properties: name (string), parent (direct relation to FlowComponent), value (float64), unit (string), sourceTag (string), quality (enum: Verified | Estimated | Manual). - `production_accounting.BalancePeriod:v1` — a reporting period and its rollup KPIs. Properties: periodStart (timestamp), periodEnd (timestamp), grossProduction (float64), netExport (float64). ### Spaces - `production_accounting` — the app's own data model space. - `cdf_cdm` — read-only; Cognite's core data model space. ``` Your agent reads `SPEC.md` before making feature decisions and keeps it in sync with code changes as directed by `AGENTS.md`. ## Spec-kit When you opt into spec-kit during `apps create`, the CLI installs a curated vendor bundle. The following list describes what is added to your repository. * **`.specify/`** — templates, bash scripts, and configuration (`branch_numbering: sequential`). * **`.claude/commands/speckit.*.md`** — slash commands for Claude Code. * **`.cursor/commands/speckit.*.md`** — slash commands for Cursor. * **`.specify/memory/constitution.md`** — a symlink to `AGENTS.md`, so spec-kit commands pick up your app's coding standards automatically. Specs are stored at `specs/-/spec.md`, one directory per feature. Numbering is sequential: `001-production-accounting`, `002-export-report`, and so on. For a full worked run of `/speckit.specify`, `/speckit.clarify`, and `/speckit.plan`, including the exact commands, the clarification questions and answers, and the generated plan, see [Plan a dashboard app with spec-kit](/cdf/flows/guides/spec-kit-walkthrough). The sections below summarize each command with a short example. ### `/speckit.specify` The `/speckit.specify` command generates a new feature spec directory from a short description. It creates `specs/-/spec.md` pre-populated with the spec-kit template, including a Flows-specific **Data Models & CDF Integration** section. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} /speckit.specify Build an Oil Balance Detail dashboard called 'Production Accounting' with a KPI bar and a Balance Reconciliation Tree of flow components with quality badges. Read data from CDF data models. React + Tailwind. ``` This example creates `specs/001-production-accounting/spec.md`. Complete the template sections before the next step, with **Data Models & CDF Integration** as the highest priority. ### `/speckit.clarify` The `/speckit.clarify` command reads the current spec, surfaces questions that would block planning, and does not generate code. The command runs an internal coverage scan (functional scope, data model, integration, edge cases, and more), then asks up to five targeted questions, each with a recommendation and options. For the Production Accounting dashboard, it surfaced questions such as: * Is the Imbalance value derived from the tree, or independent of it? * Is the Imbalance warning threshold hardcoded or user-configurable? * Which UI state, if any, should persist in the CDF URL? You answer each question in your agent session, and the answers are written directly back into the **Clarifications** section of `spec.md`, so the choices are recorded in git. ### `/speckit.plan` The `/speckit.plan` command reads the clarified spec and generates a numbered implementation plan inside the feature directory — typically a summary, a technical context block, a constitution check against your `AGENTS.md` coding standards, and a project structure: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} specs/001-production-accounting/ ├── plan.md ├── research.md ├── data-model.md ├── quickstart.md ├── contracts/ └── tasks.md # created later by /speckit.tasks ``` If you prefer not to run `/speckit.plan`, you can plan the implementation another way — for example, with your agent's in-session plan mode — and still produce a numbered implementation plan in the feature directory. When you run `/speckit.tasks`, it decomposes that implementation plan into the ordered list in `tasks.md` (see `/speckit.tasks`). The plan can come from `/speckit.plan` or from your own write-up, as long as the steps are numbered and ordered clearly enough to split into tasks. ### `/speckit.tasks` The `/speckit.tasks` command writes an ordered task list to `tasks.md` in the feature directory, next to `spec.md`, following the pattern `specs/-/tasks.md` (this guide uses `specs/001-production-accounting/tasks.md`). Each task maps back to a functional requirement so your agent can track progress against the spec. ### `/speckit.implement` The `/speckit.implement` command runs the tasks in order, using both the spec and task list as context. It works through the Production Accounting feature — scaffolding the KPI bar, building the reconciliation tree with collapsible rows and quality badges, wiring the quality filter, and integrating with the CDF host via `@cognite/app-sdk`. It stops at each task to confirm before continuing. ## Where do specs live? Specs live in your repository alongside your code. Check them in with your changes so behavior, intent, and data model choices stay traceable in the same history as the implementation. Review spec updates in pull requests and keep them versioned in git under the same ownership model you use for source files. ## Spec-driven development vs. waterfall Classic waterfall runs into trouble when design is fixed for a long time before code ships. This workflow is different: you record enough to start building, then tighten the spec as you learn from implementation. The target is a *good enough* plan — clarity without endless prework. Ten minutes capturing intent in a spec often saves hours of repeated clarification with an agent that still lacks shared context. You won't write a perfect spec, and that is expected. Unknowns stay open until you resolve them, and when requirements change, you update the document and keep going. The spec is not a contract locked at kickoff; it is a living document that lowers the cost of change as you learn. ## Next steps * [Plan a dashboard app with spec-kit](/cdf/flows/guides/spec-kit-walkthrough) — Run through specify, clarify, and plan on a real example. * [Get started with Flows](/cdf/flows/guides/getting-started) — Create an app and see `SPEC.md` land in your workspace. * [Flows skills](/cdf/flows/concepts/skills) — Agent-agnostic skills that ship alongside your spec in every Flows custom app. * [spec-kit on GitHub](https://github.com/github/spec-kit) — Upstream project and full command reference. # Integrating AI agents with Flows Source: https://docs.cognite.com/cdf/flows/guides/ai_agent_integration Step-by-step guide to integrating Atlas AI agents with Cognite Flows custom apps in Cognite Data Fusion (CDF) using @cognite/app-sdk. This guide is for developers building **Flows** custom apps in **Cognite Data Fusion (CDF)**. You will connect your app to any agent built with [Atlas AI](/cdf/atlas_ai) in CDF: expose app state as resources, register actions the agent can call, and send messages from your UI. Complete the [prerequisites](#prerequisites) before you follow the integration steps. With `@cognite/app-sdk@0.3.1` or later, you can: * **Let the agent see your Flows custom app.** Expose the app's content and state as resources the agent can add to its context. * **Let the agent interact with your Flows custom app.** Register actions as tools the agent can use to perform tasks in your app on the user's behalf. * **Send messages from your Flows custom app to the agent.** Trigger context-aware prompts when the user clicks a button or reaches a key point in a workflow. ## Prerequisites Before you start, ensure you have: * Access to **Cognite Data Fusion (CDF)** with appropriate capabilities. See [Assign capabilities](/cdf/access/guides/capabilities#flows-custom-apps) for more information. * A **Flows** development environment set up. See [Get started with Flows](/cdf/flows/guides/getting-started). * `@cognite/app-sdk@0.3.1` or later installed. This version adds the agent panel, messaging, and server registration APIs used in this guide. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm install @cognite/app-sdk ``` * A `HostAppAPI` instance from `connectToHostApp` on mount: ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { connectToHostApp } from '@cognite/app-sdk'; const { api } = await connectToHostApp({ applicationName: 'my-app' }); ``` `connectToHostApp` only works when your Flows app runs **inside CDF** (embedded in a CDF page with a parent host window). If you open only the Vite dev URL, for example, `https://localhost:3001`, without loading the app through CDF, there is no host to connect to and the promise rejects. Wrap the call in `try`/`catch`. When connect fails, skip `registerAgentServer` and other agent APIs, and hide or disable buttons that open the agent panel or send messages. See [Development and production behavior](#development-and-production-behavior) for an example. ## Integrate the agent Use your `HostAppAPI` instance to control the Atlas AI agent panel, send messages from your app, and register an agent server with the resources and actions the agent can use. Call `api.sendAgentLayoutMode` to change the visibility or layout of the agent panel. ```typescript theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} await api.sendAgentLayoutMode({ mode: 'sidebar' }); ``` **AgentLayoutPayload options:** | **Field** | **Type** | **Description** | | - | - | - | | `mode` | `'sidebar' \| 'fullscreen' \| 'closed'` | Target panel layout. | | `conversationId` | `string?` | Resume a specific existing conversation. | | `agentExternalId` | `string?` | Target a specific registered agent. | | `source` | `AgentSource?` | Surface identifier; omit unless required by your host. | `{ mode: 'sidebar' }` is the most common call, typically wired to a persistent toolbar button that opens the assistant at any time. Call `api.sendAgentMessage` to inject text into the agent's chat, optionally starting a fresh session. Use `sendAgentMessage` to pre-populate the agent with context when the user clicks a contextual trigger in your app. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} await api.sendAgentMessage({ message: 'Summarize the current state and highlight any risks.', newSession: true, }); ``` **AgentMessagePayload options:** | **Field** | **Type** | **Description** | | - | - | - | | `message` | `string` | Text to inject into the chat. | | `newSession` | `boolean?` | Clear the existing conversation before injecting. | Call `sendAgentLayoutMode` before `sendAgentMessage`. The agent panel must be open for the user to see the injected message. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} await api.sendAgentLayoutMode({ mode: 'sidebar' }); await api.sendAgentMessage({ message: `Analyze the schedule for "${tarName}" and suggest how to reduce the total duration.`, newSession: true, }); ``` Use `newSession: true` for contextual entry points where the user is starting a new task. Omit it when continuing an existing conversation. An agent server exposes resources (read-only context the agent can request) and actions (tools the agent can call). Register it once on mount, and unregister it on unmount. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { createAgentServer, registerAgentServer } from '@cognite/app-sdk'; const server = createAgentServer({ uri: 'my-app://agent-server-unique-id', // namespaced by CDF with the instance ID actions: [...], resources: [...], }); await registerAgentServer(api, server); // On unmount: await api.unregisterAgentServer('my-app://agent-server-unique-id'); ``` In React, manage the lifecycle with `useEffect`: ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} useEffect(() => { if (!api) return; const server = createAgentServer({ uri: 'my-app://agent-server-unique-id', actions, resources }); void registerAgentServer(api, server); return () => { void api.unregisterAgentServer('my-app://agent-server-unique-id'); }; }, [api]); ``` `uri` must be unique within the current app. Different apps can reuse the same `uri`; CDF namespaces it with the instance ID. If the same app registers another server with that `uri`, the new server replaces the existing one. Resources are read-only data the agent reads to understand the current app state. Each resource has a URI, a name, and a `read` function that returns content. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { createAgentResource } from '@cognite/app-sdk'; createAgentResource({ uri: 'my-app://current-state', name: 'Current application state', description: 'Describes what data the agent will receive and when it should read this resource.', async read() { const data = getStateFromSomewhere(); return [{ type: 'json', data }]; }, }) ``` The `read` function returns an array of content parts. Each element is one of: * `{ type: 'json', data: unknown }` — structured data. The agent reasons over JSON directly; prefer this format when you can. * `{ type: 'text', text: string }` — free-form text. The `description` field is visible to the agent. Use it to explain what the resource contains and when to read it. Treat it like a function docstring. ### Example: TAR planner resources The TAR planner application exposes two resources. Both are populated from `TARStorageService` (localStorage) and the critical path algorithm, giving the agent a live view of what the user is managing. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} createAgentResource({ uri: 'tar-planner://current-tar', name: 'Current TARs', description: 'All TARs with name, facility, scope, dates, status, and work order count.', async read() { return [{ type: 'json', data: storage.getAllTars() }]; }, }); createAgentResource({ uri: 'tar-planner://schedule-summary', name: 'Schedule summary', description: 'Per-TAR computed schedule with critical path flags and float.', async read() { return [{ type: 'json', data: computeScheduleSummary(storage) }]; }, }); ``` Actions are tools the agent can invoke. They can be read-only queries or mutating operations. Define parameters with a Zod schema. The `.describe()` call on each field is the agent's documentation for that parameter. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { createAgentAction } from '@cognite/app-sdk'; import { z } from 'zod'; createAgentAction({ name: 'tar-planner://get_item_details', // snake_case; shown to the agent as the tool name description: 'What this does, what it returns, and when to call it.', parameters: z.object({ item_id: z.string().describe('The ID of the item to retrieve'), include_history: z.boolean().optional().describe('Whether to include audit history'), }), async handler({ item_id, include_history }) { const item = await myService.getItem(item_id, { history: include_history }); return { content: [{ type: 'json', data: item }], }; }, }) ``` ### Mutating actions Mutating actions can write to your app's state. The agent does not currently confirm with the user before calling mutating actions. Use mutating actions with caution and make the agent's responsibility explicit in the action's `description`. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} createAgentAction({ name: 'tar-planner://add_work_order_to_tar', description: 'Add a maintenance order to a TAR. Call this ONLY when the user has explicitly approved. The app UI updates immediately.', parameters: z.object({ tar_id: z.string().describe('The TAR to add the work order to'), maintenance_order_external_id: z.string().describe('The externalId of the order'), }), async handler({ tar_id, maintenance_order_external_id }) { storage.saveEntry({ tarId: tar_id, maintenanceOrderExternalId: maintenance_order_external_id, ... }); return { content: [{ type: 'json', data: { success: true } }] }; }, }) ``` After a mutating action writes to your state layer, the UI updates on the next render cycle if your components read from the same state. No extra signaling is required. ## Verify integration Load your Flows custom app **inside CDF** (through the CDF development URL, not only `https://localhost:3001`). Confirm the following: 1. Open the agent panel from your UI control or a `sendAgentLayoutMode` call. 2. Send a test message and confirm it appears in the agent chat. 3. Ask the agent a question that should use a registered resource and confirm the answer reflects your app state. 4. If you registered mutating actions, test them only after you understand the risks in [Mutating actions](#mutating-actions). When integration works, the agent panel opens from your app, messages appear in chat, and resource-backed answers match the state your app exposes. ## Recommended project structure Separate the registration lifecycle from the action and resource definitions to test both independently. ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} src/features/agent/ agentActions.ts — pure factory: (deps) => Action[] agentResources.ts — pure factory: (deps) => Resource[] useAgentServer.ts — useEffect lifecycle hook; calls the factories and registers ``` Inject services as parameters rather than importing them directly. That approach lets you unit test actions and resources with mock services. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} // agentActions.ts export function buildAgentActions( storage: StorageService, dataService: DataService, ): AgentActions { return [ createAgentAction({ ..., async handler({ id }) { return storage.get(id); } }), ]; } // useAgentServer.ts export function useAgentServer(api: HostAppAPI | null): void { const storage = useStorageService(); // injected via React context const dataService = useDataService(); useEffect(() => { if (!api) return; const server = createAgentServer({ uri: 'tar-planner://planning-server', actions: buildAgentActions(storage, dataService), resources: buildAgentResources(storage, dataService), }); void registerAgentServer(api, server); return () => { void api.unregisterAgentServer('tar-planner://planning-server'); }; }, [api, storage, dataService]); } ``` To test `buildAgentActions`, pass mock service implementations and call `handler` directly without mounting the hook. ## Development and production behavior `connectToHostApp` behaves differently depending on where your app runs: | **Environment** | **`connectToHostApp` behavior** | **Effect** | | - | - | - | | Inside CDF | Resolves with `{ api }` | Full agent integration functionality. | | Standalone development (`vite dev`) | Rejects | Agent features disabled; hide or disable agent UI triggers. | Use the following pattern to handle both environments: catch rejections from `connectToHostApp` and disable agent features when your app isn't running inside CDF. ```typescript wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} let api: HostAppAPI | null = null; try { ({ api } = await connectToHostApp({ applicationName: 'my-app' })); } catch { // Not running inside CDF: agent features disabled } ``` In React, hold `api` in state or context. When the host connection succeeds, your registration effect runs and UI components can show or hide agent triggers. ## Further reading * [Get started with Flows](/cdf/flows/guides/getting-started) * [Flows authentication](/cdf/flows/reference/api/auth) * [About Atlas AI agents](/cdf/atlas_ai/concepts) * [Atlas AI agent building](/cdf/atlas_ai/guides/atlas_ai_agent_building) * [Using Atlas AI agents via API and SDK](/cdf/atlas_ai/guides/atlas_ai_agent_api_sdk) * [CDF SDK documentation](/dev/sdks/) # Certifying a Flows custom app for production Source: https://docs.cognite.com/cdf/flows/guides/application-certification Run the four skill-driven steps that certify a Flows custom app to publish it to a production environment. **Application certification** authorizes a specific Flows custom app version for a production environment. The application certification signature is required for production publish in addition to your developer signature. The process confirms the app meets the same security, data, and UX standards that builders are trained against in [Builder certification](/cdf/flows/guides/builder-certification), and it applies to every app independently. Application certification is required for **every** Flows custom app published to a production CDF project, even when built by a certified builder. Each app version is certified independently, and **every code change** (new feature, bug fix, refactor, dependency bump) requires re-certification before you deploy and publish a new version to production. ## When you need it | You're publishing to ... | Application certification | | - | - | | A **production** project, first publish | **Required** before publish | | A **production** project, **any code change** (feature, bug fix, refactor, dependency bump) | **Required** before publishing a new version to production | | A CDF project with [**Dev status**](/cdf/flows/guides/dev-status) | Not required. You still need a developer signature before you publish. See [Code signing](/cdf/flows/guides/code-signing). | [Request Dev status](/cdf/flows/guides/dev-status) before you publish to a non-production project. **Prerequisites**: * Active [Builder certification](/cdf/flows/guides/builder-certification) (valid, not expired). * Enterprise license to **Cursor**, **Claude Code**, or another agentic coding tool (required to run the certification skills). * A Flows custom app created with `npx @cognite/cli@latest apps create`. **The flow is skill-driven.** Each step is a slash command in your agentic coding tool. The skills generate artifacts in your app's repository, and the final submit skill verifies that every prior artifact exists before packaging the build for submission. ## How the certification flow works Run `/flows-app-brief` immediately after creating the app, **before** you start building. The skill acts as a certification coach: it pre-scans `app.json` and any spec files for context, then asks targeted questions about your user, the problem the app solves, the one-sentence value statement, and how you'll measure success. It writes `App-Brief.md` at the repo root. `App-Brief.md` is the user-context document reviewers read first. The final submit skill blocks submission until every required field is populated. Develop your app using the latest Flows template. Follow the [Get started with Flows](/cdf/flows/guides/getting-started) guide for setup. Pull skills as you go (`npx @cognite/cli@latest apps skills pull`) so AI coding agents stay aligned with current Cognite best practices. Run `/flows-code-review` to perform the technical review. The skill produces three artifacts under `reviews/code-review/feedback-round-N/`: * `code-review-report.md` — scored report covering correctness, SDK usage, dependencies, test coverage, dead code, patterns, Data Modeling Service (DMS) query efficiency, pagination, rate limiting, and Aura usage. * `review-files.md` — per-file inventory. * `review-packages.md` — dependency audit including `npm audit` results. The report's `Summary` block reports a `Must Fix open:` count. Re-run the skill (it creates a new `feedback-round-N`) until the count is **0**. Run `/flows-design-review` to score the app against the [10 design quality questions](/cdf/flows/guides/quality-guidelines). The skill is semi-automated: for each question it runs concrete repo probes (grep, lint, build) and proposes a draft 1–5 score. You confirm or override each score after walking your tasks end-to-end in the running app. It writes `design-review-report.md` under `reviews/design-review/feedback-round-N/`. Target overall average: **3.8 or higher**. Before running this skill, ensure you have run: * `npx @cognite/cli@latest apps deploy` — builds the app automatically (`npm run build`) and uploads it, creating the deploy bundle at `.cognite-bundles/-.zip` * `npx @cognite/cli@latest apps sign` — signs the bundle and writes `.dev.sig` Ensure your working tree is clean (`git status` shows nothing uncommitted) before running `apps deploy`. The build runs automatically and the source archive must match the deployed bundle exactly. See [Deploy a Flows custom app](/cdf/flows/guides/deploying) for build options. Run `/flows-external-app-submit`. The skill verifies five gates before packaging anything: | Gate | Required | | - | - | | `App-Brief.md` complete (required fields: `appName`, `customer`, `tier`, `owner`, `userRole`, `currentProblem`, `oneSentenceStory`, `successCriteria`, `userEvidence`) | Yes | | Latest `code-review-report.md` reports `Must Fix open: 0` | Yes | | Latest `design-review-report.md` reports `Average score: 3.8` or higher | Yes | | No secrets in tracked files (`.env*`, `.pem`, `.key`, `.p12`, etc.) | Blocked | | Working tree clean | Blocked | On all-pass, the skill runs `npx @cognite/cli@latest apps submit`, which collects the source archive (`src-.zip`) and the deploy bundle into `dist/certification/`, then prompts you to add your screen recording to that folder before opening the pre-filled Zendesk form. Open the [Cognite Zendesk submission form](https://cognite.zendesk.com/hc/en-us/requests/new?ticket_form_id=360001234312\&tf_priority=normal\&tf_subject=App+Review+Request%3A+%5BApp+name%5D\&tf_360015295097=cognite_flows\&tf_description=Dear+Cognite+Platform+Team%2C%3Cbr%3E%3Cbr%3EI+would+like+to+submit+my+Flows+app+for+review+and+certification.%3Cbr%3E%3Cbr%3EApp+name%3A+%5BApp+name%5D%3Cbr%3EApp+external+ID%3A+%5BexternalId%5D%3Cbr%3EVersion%3A+%5BversionTag%5D%3Cbr%3E%3Cbr%3E%60npx+%40cognite%2Fcli+apps+submit%60+verified+App-Brief.md%2C%3Cbr%3Ecode-review-report.md%2C+and+design-review-report.md+are+committed%3Cbr%3Eand+present+in+the+source+archive.%3Cbr%3E%3Cbr%3EAttached%3A%3Cbr%3E%5B+%5D+Source+archive+%28.zip%29+generated+by+%60npx+%40cognite%2Fcli+apps+submit%60%3Cbr%3E%5B+%5D+Screen+recording+of+the+application+%28required+for+design+review%29%3Cbr%3E%3Cbr%3EBest+regards%2C%3Cbr%3E%5BYour+name%5D). The form, subject, description, and **Cognite Flows** product field are pre-filled. Update the placeholders with your app's details. Place your screen recording in `dist/certification/` (the CLI prompts you to do this), then attach all files from that folder: * **`src-.zip`** — source archive (committed code only; `design-review-report.md` is included inside) * **`-.zip`** — deploy bundle; required for signature verification * **Screen recording** of the application walking through the primary tasks (required for the design review) Submit the ticket. Your submission is reviewed manually. You'll receive one of two outcomes from the Zendesk review: * **Feedback:** Address reviewer comments, re-run the relevant skills, commit the updated artifacts, and reply in the same Zendesk ticket with a fresh `dist/certification/` folder (new `src-.zip`, deploy bundle, and updated screen recording if the UI changed). * **Approval:** Download the `.cert.sig` from the [Yopass](https://yopass.se/) link in the ticket. Place it alongside your `.dev.sig` file, run `npx @cognite/cli@latest apps publish`, then `npx @cognite/cli@latest apps activate`. No committed SLA at launch. Reviews are first-in, first-out as the team scales the process. ## App tiers The App Brief asks you to pick a tier so reviewers can calibrate expectations. Pick the closest match; "not sure" is fine, and the reviewer can adjust. | Tier | Scope | Examples | | - | - | - | | **Tier 1: Monitoring & reporting** | Read-only dashboards, asset/equipment overviews, KPI reports | Asset health dashboard, production KPI overview | | **Tier 2: Operational support** | Interactive workflows that support an operator's day, typically single user group | Work-order triage, data validation, shift handover | | **Tier 3: Business critical** | Multi-step workflows, multiple roles, write-back into systems of record | Approval workflows, multi-user collaboration with state changes | ## Submission requirements | Item | Source | | - | - | | Source archive `src-.zip` | `dist/certification/` — created by `npx @cognite/cli@latest apps submit` | | Deploy bundle `-.zip` | `dist/certification/` — copied from `.cognite-bundles/` by `apps submit`; must match the signed hash | | Screen recording of the application | `dist/certification/` — placed there by the builder; should cover the primary user tasks named in `App-Brief.md` | | `App-Brief.md` (inside `src-.zip`) | Generated by `/flows-app-brief` | | Latest `code-review-report.md` (inside `src-.zip`) | Generated by `/flows-code-review` | | Latest `design-review-report.md` (inside `src-.zip`) | Generated by `/flows-design-review` | ## Self-check before submitting Run this checklist immediately before you open the Zendesk ticket: * [ ] The app is targeting a **production** project without [Dev status](/cdf/flows/guides/dev-status) (skip certification for Dev status projects). * [ ] `App-Brief.md` exists at the repo root and every required field is populated. * [ ] The latest `reviews/code-review/feedback-round-N/code-review-report.md` reports `Must Fix open: 0`. * [ ] The latest `reviews/design-review/feedback-round-N/design-review-report.md` reports `Average score:` of **3.8** or higher. * [ ] All skill-generated artifacts (`App-Brief.md` and the latest `reviews/` reports) are committed to git (`git add` + `git commit`); the submit skill packages only committed files. * [ ] `npx @cognite/cli@latest apps deploy` has been run; deploy bundle exists at `.cognite-bundles/`. * [ ] `npx @cognite/cli@latest apps sign` has been run; `.dev.sig` exists. * [ ] `/flows-external-app-submit` ran and printed PASS for every gate. * [ ] Screen recording of the app's primary tasks has been placed in `dist/certification/`. * [ ] All files in `dist/certification/` (`src-.zip`, deploy bundle, screen recording) are ready to attach. If every box is ticked, the submission is ready. Submit through the [Cognite Zendesk submission form](https://cognite.zendesk.com/hc/en-us/requests/new?ticket_form_id=360001234312\&tf_priority=normal\&tf_subject=App+Review+Request%3A+%5BApp+name%5D\&tf_360015295097=cognite_flows\&tf_description=Dear+Cognite+Platform+Team%2C%3Cbr%3E%3Cbr%3EI+would+like+to+submit+my+Flows+app+for+review+and+certification.%3Cbr%3E%3Cbr%3EApp+name%3A+%5BApp+name%5D%3Cbr%3EApp+external+ID%3A+%5BexternalId%5D%3Cbr%3EVersion%3A+%5BversionTag%5D%3Cbr%3E%3Cbr%3E%60npx+%40cognite%2Fcli+apps+submit%60+verified+App-Brief.md%2C%3Cbr%3Ecode-review-report.md%2C+and+design-review-report.md+are+committed%3Cbr%3Eand+present+in+the+source+archive.%3Cbr%3E%3Cbr%3EAttached%3A%3Cbr%3E%5B+%5D+Source+archive+%28.zip%29+generated+by+%60npx+%40cognite%2Fcli+apps+submit%60%3Cbr%3E%5B+%5D+Screen+recording+of+the+application+%28required+for+design+review%29%3Cbr%3E%3Cbr%3EBest+regards%2C%3Cbr%3E%5BYour+name%5D). ## Troubleshooting common issues Open the latest `reviews/code-review/feedback-round-N/code-review-report.md` and work the `Must Fix` section top to bottom. Re-run `/flows-code-review` after each batch of fixes; it creates a new feedback round so reviewers can see your progression. Submission stays blocked until the most recent round reports `Must Fix open: 0`. The skill proposes scores from repo probes and your walkthrough notes; you can override any score it gets wrong. Re-run `/flows-design-review` in a new feedback round, override the scores you disagree with, and add an `Improvement note` explaining the reasoning so reviewers can follow your thinking. Any CDF project that does **not** have [Dev status](/cdf/flows/guides/dev-status) and is used for production decisions, dashboards, or workflows. Typically the project with `production` (or similar) in the name. If you're unsure, ask the project's CDF administrator. When in doubt, certify, or [request Dev status](/cdf/flows/guides/dev-status) for non-production projects. No. The design review needs to see the app in motion across your primary user tasks. A short walkthrough (2–5 minutes) is enough; narration is optional but helpful. Yes. Application certification is **per-version** and applies to every code change (new feature, bug fix, refactor, or dependency bump). Re-run the four skills against the new version and submit a fresh zip referencing the new commit hash before you republish to production. Yes. Certification is per-commit, not per-change-size. The re-run is fast when nothing meaningful has changed (the code-review and design-review skills produce reports quickly against a known-good codebase), so the friction is small. The guarantee that production only ever runs certified code is what matters. Make the fixes, re-run `/flows-code-review` and `/flows-design-review` (each in a new feedback round), then re-run `/flows-external-app-submit` to produce a fresh zip. Open a new Zendesk ticket and note in the description that this is a re-certification, referencing the original ticket number. ## Further reading * [Builder certification](/cdf/flows/guides/builder-certification) — Certify yourself before you certify an app. * [Dev status](/cdf/flows/guides/dev-status) — Skip application certification on non-production projects. * [Quality guidelines](/cdf/flows/guides/quality-guidelines) — The 10 design quality questions the design review skill scores against. * [Skills](/cdf/flows/concepts/skills) — How skills are delivered and refreshed in your app. * [Get started with Flows](/cdf/flows/guides/getting-started) — Scaffold a new app. # Becoming a certified Flows custom apps builder Source: https://docs.cognite.com/cdf/flows/guides/builder-certification Learn how to complete the builder certification to develop and deploy Flows custom apps. **Builder certification** qualifies a developer (internal, partner, or customer) to build production-ready Flows custom apps in customer **Cognite Data Fusion (CDF)** projects, including non-production projects. It confirms you've met security, data, and UX standards before working on customer projects. Builder certification authorizes you as a builder. Each app you publish to a production CDF project still needs [application certification](/cdf/flows/guides/application-certification). ## Certification scope and validity * **Who it covers**: Individual builders (internal, partner, or customer) who complete the Flows custom apps training path. * **What it covers**: Authorization to build Flows custom apps in customer CDF projects. * **Validity**: Six months from the date you claim the certificate. ## Prerequisites **Flows custom apps** is a pro-code product: you write and maintain the application code. Before you start, confirm that you have: * Proficiency with an agentic coding tool. [Cursor](https://cursor.com/) or [Claude Code](https://www.anthropic.com/claude-code) is strongly preferred. * Access to a CDF project where you can build. A development or test project is fine. * Working familiarity with CDF, the Cognite JavaScript Software Development Kit (SDK), and CDF data modeling concepts. See [Getting started with Flows custom apps](/cdf/flows/guides/getting-started) for project setup. ## Completing the certification path Internal, partner, and customer builders follow the same path: 1. Complete [Cognite Flows Foundation](https://hub.cognite.com/p/cognite-flows). 2. Attend a one-day hands-on workshop. 3. Claim the Cognite Flows builder certificate. 4. Generate and register a signing key. Before the enablement session, finish: * The [Cognite Flows Foundation](https://hub.cognite.com/p/cognite-flows) course. * The persona-matched upskilling material your Cognite representative sends you (depending on whether you're a solution architect, app developer, dashboard creator, platform owner, or operational subject matter expert (SME)). Join the on-site Flows enablement session. The session facilitator reviews the theory at the start. Then you build against live CDF data on real use cases, using the certification skills (`/flows-app-brief`, `/flows-code-review`, `/flows-design-review`) end to end. Start with `/flows-app-brief` after you create the app with `npx @cognite/cli@latest apps create`. It produces `App-Brief.md`, which [application certification](/cdf/flows/guides/application-certification) requires later. Once you've completed the hands-on workshop, claim your certificate and digital badge. After you claim the certificate, you're a certified Flows custom apps builder for six months. After you claim the certificate, generate a signing key and register the public key with Cognite. You need this key to publish apps. The private key never leaves your machine. Follow [Generate and register a signing key](/cdf/flows/guides/code-signing#generate-and-register-a-signing-key). Cognite support checks your **Cognite Flows Foundation** completion and the hands-on workshop before adding your public key. When the key is in the registry, use [Upload, sign, and publish the app](/cdf/flows/guides/code-signing#upload-sign-and-publish-the-app) to publish each version. ## Recertification To maintain active status, recertify before your six-month certification expires. Recertification is self-serve: retake the latest Academy course and refresh your hands-on work against the current Flows custom apps template and skills. ## Troubleshooting common issues No, attendance alone isn't enough. Finish the Academy pre-work and the hands-on building, then claim your certificate. You're a certified builder when you claim the certificate. For initial certification, no. Internal, partner, and customer builders go through the Flows enablement session as the single supported path. **Recertification** is self-serve once your initial certification is in place. No. The app you build during the enablement session must operate against live CDF data so the certification skills produce meaningful reports and the session facilitator can assess your work properly. No. Builder certification authorizes you to build Flows custom apps in production environments. Each app you publish to a **production** project still needs [application certification](/cdf/flows/guides/application-certification) so Cognite can issue the app certification signature (`.cert.sig`). Projects with [Dev status](/cdf/flows/guides/dev-status) skip this signature, but you must still add a developer signature before you publish. See [Signatures required to publish](/cdf/flows/guides/code-signing#signatures-required-to-publish). Recertify by retaking the latest Academy course and refreshing your hands-on work. Contact your Cognite representative if you've been out of practice for an extended period and want a refresher session. ## Further reading * [Application certification](/cdf/flows/guides/application-certification) — Required for every Flows custom app published to a production project. * [Dev status](/cdf/flows/guides/dev-status) — Request Dev status to publish builder-signed apps on a non-production project without application certification. * [Code signing](/cdf/flows/guides/code-signing) — Sign bundles with a developer signature. * [Flows custom apps quality guidelines](/cdf/flows/guides/quality-guidelines) — The 10 design quality questions the design review skill scores against. * [Builder skills](/cdf/flows/concepts/skills) — How the certification skills are delivered and refreshed. * [Getting started with Flows custom apps](/cdf/flows/guides/getting-started) — Scaffold a new app and set up your project. * [Building Flows custom applications](/cdf/flows) — Template capabilities and tooling overview. # Signing Flows custom apps from the CLI Source: https://docs.cognite.com/cdf/flows/guides/code-signing Step-by-step guide to generate signing keys, sign Flows custom app bundles, and publish signed deployments with the Cognite CLI. Signing a Flows custom app proves who built the bundle and binds the signature to the exact bytes deployed. The platform rejects the version if anything in the bundle changes after signing, or if the signing key isn't registered. This guide has two parts. First, generate and register a signing key. Then upload, sign, and publish each version of the app. Use this guide if you're a certified Flows builder and you publish custom apps with the Cognite CLI. ## Signatures required to publish Every publish needs a **developer signature**. Projects with [**Dev status**](/cdf/flows/guides/dev-status) need only that signature. Production **Cognite Data Fusion (CDF)** projects without Dev status need **both** that signature and Cognite's **app certification signature** (`.cert.sig`) from [Application certification](/cdf/flows/guides/application-certification). Each signature is tied to the exact bundle bytes. This guide covers the developer signature only. | Signature | What it proves | | - | - | | Developer signature | A certified developer built this bundle for this CDF project. | | App certification signature | Cognite has reviewed and approved this production bundle. | [Request Dev status](/cdf/flows/guides/dev-status) before you publish to a non-production project. ## Prerequisites * The latest Cognite CLI (`npx @cognite/cli@latest`). * A Flows custom app with `app.json` (with `infra` set to `appsApi`). * Access to the CDF project you'll deploy to. * [Builder certification](/cdf/flows/guides/builder-certification), or signing instructions from the team reviewing your app. * [Application certification](/cdf/flows/guides/application-certification) if you're publishing to a production project and need Cognite's `.cert.sig`. A developer signature is not the same as [application certification](/cdf/flows/guides/application-certification). Production apps still need Cognite's application certification signature before they can be published to production projects. ## Generate and register a signing key Do this after you claim the [builder certificate](/cdf/flows/guides/builder-certification). Every publish needs a developer signature from a key that Cognite has registered. Generate a signing key and submit the Zendesk URL that the CLI prints. You need one registered key to publish apps to any CDF project. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest keys generate --interactive ``` When prompted for a key ID (`kid`), give the key a short name for this machine, such as `juno-laptop`. The key expires in **12 months**. On macOS, the private key goes in **Keychain** (a password prompt is expected). On Windows, it goes in **Credential Manager**. Don't email the `.pub.pem` file. The Zendesk URL already includes the public key. These flags change how the CLI collects the name and how you get the Zendesk URL. | Flag | What it does | | - | - | | `--interactive` | Prompts you to **Open in browser** or **Copy link**. Opens a pre-filled Zendesk form. Does not submit the form. | | `--copy-link` | Copies the Zendesk URL with no prompt. | | `--kid` | Sets the key ID. Use a short name for this machine. | | `--email` | Sets the email address on the Zendesk form. | If you pass `--kid` and `--email` together, the CLI skips prompts and prints the URL plus a YAML block. Use the URL. Don't paste YAML into a blank ticket. The private key stays on your machine. Don't share it, commit it, export it from Keychain, or send it to Cognite. Submit the Zendesk form from the URL that `keys generate` printed. Don't paste YAML into a blank ticket. Support checks that you completed **Cognite Flows Foundation** and the **hands-on workshop**, then replies when they add the key. New keys become available on weekdays during business hours in Central European Time (CET). Until then, `apps status` can show `KEY_NOT_IN_REGISTRY`. If **publish** fails with a 403 that mentions CDF capabilities, that message is generic. It is not an access problem. See [403: signing key is not in the public key registry](#403-signing-key-is-not-in-the-public-key-registry). When support has added the key and it is available, you can publish apps with a developer signature. ## Upload, sign, and publish the app After your public key is registered, use these steps to upload, sign, and publish each version of the app. From your app folder: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run build npx @cognite/cli@latest apps deploy --interactive ``` `apps deploy` packages `dist/`, uploads it as a DRAFT, and keeps a copy of the exact bytes at `.cognite-bundles/-.zip` so signing operates on what the platform actually received. The app scaffold gitignores `.cognite-bundles/`. Don't commit bundle files from that folder. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps sign ``` With no arguments, `apps sign` uses `externalId` and `versionTag` from `app.json`. It finds the matching `.cognite-bundles/-.zip`, writes `.dev.sig` next to it, and submits the signature to the platform. Override with `--signing-identity `, `--scope /`, or an explicit bundle path. Run `apps status` to verify the signature was accepted before you publish. Commit before signing so the source-to-signature link stays auditable. Tag the commit with the app version: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} git tag "v$(jq -r .versionTag app.json)" -m "Signed deployment" git push origin --tags ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps publish . ``` `apps publish` transitions the version from DRAFT to PUBLISHED. Signatures were already submitted by `apps sign`; publish re-submits any local `.dev.sig` (and `.cert.sig` if present) as a safety net, then flips the lifecycle state. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps status . ``` The command shows the lifecycle state and every signature the platform has on record for the current version. Each entry includes the signing identity (`kid`), the role (`developer` or `certifier`), the time the signature was issued, and the platform's verdict. You'll see one of three shapes depending on where you are in the flow: Before `apps deploy`: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Status: not deployed yet Run `npx @cognite/cli apps deploy` to upload this version. ``` After `apps deploy`, before `apps sign` + `apps publish`: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Status: DRAFT Signatures: none stored Run `npx @cognite/cli apps publish .` to publish this version. ``` After a clean publish: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} App: Sign Test App (sign-test-app) Version: 0.0.3 (local) Status: PUBLISHED Signatures: acme-dev-001 (developer, signed 2026-05-27T13:22:37.000Z) — VALID cognite-cert-1 (certifier, signed 2026-05-27T13:25:10.000Z) — VALID ``` A verdict other than `VALID` means the platform won't accept that signature for serve-time verification. The verdict table explains each result and what to do. | Verdict | What it means | What to do | | - | - | - | | `VALID` | Signature verifies and the key is in the registry and in date. | Nothing — this is the expected state. | | `REVOKED` | The signing key has been revoked since the signature was submitted. | Re-sign with a current key, then publish a new version. | | `EXPIRED` | The signing key was in date when signed, but has since expired. | Rotate the key (see [Rotate or replace a key](#rotate-or-replace-a-key)) and re-sign. | | `SIGNED_BEFORE_KEY_ISSUED` | The signature's `iat` timestamp predates the key's registration. Clock skew, or a stale key. | Re-sign with the current system clock. If the clock is correct, contact Cognite. | | `IAT_IN_FUTURE` | The signature's `iat` timestamp is in the future. Usually clock skew on the signing machine. | Fix the signing machine's clock, then re-sign and re-publish. | | `BUNDLE_TOO_OLD` | The bundle's signature is older than the platform's minimum-age policy. | Re-sign the current bundle and re-publish. | | `KEY_NOT_IN_REGISTRY` | The `kid` isn't in the public key registry, or isn't available yet. | Confirm the `kid` with `keys list`. Wait until the key is available on a weekday during business hours, Central European Time. See [403: signing key is not in the public key registry](#403-signing-key-is-not-in-the-public-key-registry). | Verify the version is **PUBLISHED** and your developer signature appears as `VALID` for the expected `kid`. Anything else needs action before the version will serve. ## Production certification (Cognite signature) Production CDF projects without [Dev status](/cdf/flows/guides/dev-status) need both a developer signature and an app certification signature. The two signatures are independent. Cognite signs the same bundle bytes with their own key and returns the signature to you. Complete the [Upload, sign, and publish the app](#upload-sign-and-publish-the-app) steps through **Sign the bundle**, then add these steps before you publish the application: From your app folder, on a clean commit: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps submit ``` `apps submit` packages a source archive (`src-.zip`), copies the deploy bundle into `dist/submit/`, and opens a pre-filled Zendesk request with the app metadata. Attach everything in `dist/submit/` plus a screen recording of the app, then submit. The command will fail if your working tree has uncommitted changes — the source archive has to match the committed code exactly. Commit first, then re-run. After Cognite reviews and approves the bundle, your Zendesk contact replies with a `.cert.sig` file. Save it to your app's `.cognite-bundles/` folder next to the deploy bundle, named `-.cert.sig`. For example, for app `sign-test-app` version `0.0.3`: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} .cognite-bundles/ sign-test-app-0.0.3.zip ← created by apps deploy sign-test-app-0.0.3.zip.dev.sig ← created by apps sign sign-test-app-0.0.3.zip.cert.sig ← from Cognite, save here ``` To open `.cognite-bundles/` in your file manager from the app folder: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} open .cognite-bundles ``` ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} explorer .cognite-bundles ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} xdg-open .cognite-bundles ``` Don't commit `.cert.sig` to git. The scaffold gitignores `.cognite-bundles/` for this reason. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps publish . ``` `apps publish` picks up both `.dev.sig` and `.cert.sig`, submits the certifier signature, and transitions the version to PUBLISHED. Run `apps status .` to confirm both signatures show as `VALID`. ## Sign again when the bundle changes The signature is bound to the exact bytes of the bundle. Anything that changes the bundle invalidates the signature: * Re-running `npm run build` * Editing files in `dist/` or `manifest.json` * Bumping the app version Re-run `apps deploy`, `apps sign`, and `apps publish` in order. `deploy` overwrites the bundle file at the same path, so the next `sign` picks up the new bytes automatically. ## Rotate or replace a key Generate a new key when the old one expires, you move to a new machine, or you suspect the private key is compromised: 1. `npx @cognite/cli@latest keys generate --interactive`. 2. Register the new public key. 3. Sign new bundles with the new key. 4. Ask your Cognite contact to revoke the old key if needed. Previously published apps stay published. Rotating your local key doesn't invalidate signatures already accepted by the platform. ## Troubleshooting Run `npx @cognite/cli@latest keys generate --interactive`, then retry the signing command. The bytes you signed don't match what the platform received. Run these commands in order: 1. `npx @cognite/cli@latest apps deploy` — refreshes `.cognite-bundles/-.zip` with the bytes the platform has. 2. `npx @cognite/cli@latest apps sign` — signs the refreshed bundle. 3. `npx @cognite/cli@latest apps publish .` — submits the new signature. Don't manually repackage the bundle. `apps sign` stores the signature even if the key isn't in the registry yet. Wait until the key is available on a weekday during business hours, Central European Time. Don't sign or publish again while you wait. `apps status` flips to `VALID` on its own. If **publish** returned 403, see [403: signing key is not in the public key registry](#403-signing-key-is-not-in-the-public-key-registry) and run `apps publish` again after the key is available. If **publish** fails with: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} signature (role=developer, kid=jsmith-dev-001): signing key used to create the signature is not in the public key registry. | code: 403 | X-Request-ID: ... You don't have the required CDF capabilities. Please contact your CDF admin. ``` The first line is the cause: that `kid` isn't in the public key registry. The second sentence is a generic message the CLI shows for any 403. It is not an access problem. Don't contact your CDF administrator for this error. Either you signed with a **different local key** than the one you registered, or the registered key **is not available yet**. Run `npx @cognite/cli@latest keys list`. If you signed with the wrong key, sign again with `--signing-identity ` and publish again. If you used the registered key, wait until the key is available on a weekday during business hours, Central European Time (CET), then run `apps publish` again. You don't need to sign again. If the error continues during business hours, Central European Time, contact Cognite support. The signature timestamp doesn't line up with the key's registration window. Almost always clock skew on the signing machine. Make sure your system clock is correct, then re-run `apps sign` and `apps publish`. You're targeting a production project that requires both a developer and an app certification signature. Follow [Production certification](#production-certification-cognite-signature) to submit for review and add the `.cert.sig` Cognite returns. Run `npx @cognite/cli@latest keys list`, find the correct `kid`, and pass it with `--signing-identity `. Generate a new key on the replacement machine and register the public key. Ask your Cognite contact to revoke the old key if it may be compromised. ## Related topics * [Becoming a certified Flows custom apps builder](/cdf/flows/guides/builder-certification) * [Certifying a Flows custom app for production](/cdf/flows/guides/application-certification) * [Deploying Flows custom apps](/cdf/flows/guides/deploying) * [Dev status](/cdf/flows/guides/dev-status) * [Flows custom apps quality guidelines](/cdf/flows/guides/quality-guidelines) # Deploying Flows custom apps Source: https://docs.cognite.com/cdf/flows/guides/deploying Deploy your Flows custom app to Cognite Data Fusion (CDF) with interactive or automated deployment and App Hosting lifecycle commands. You can deploy your Flows custom app to CDF **interactively** from your machine (browser sign-in, no OAuth secrets in the terminal) or **automate** deployment in CI/CD with client credentials. ## Interactive deployment The simplest path uses your browser for sign-in—no OAuth configuration in the terminal: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps deploy --interactive ``` This will: 1. Build your app. 2. Open a browser window for you to sign in to CDF. 3. Prompt you to select a deployment target from `app.json`. 4. Package and deploy to your configured project. 5. Show a link to your deployed app. ### Options ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Use the first deployment target (skip selection prompt) npx @cognite/cli@latest apps deploy --interactive -d 0 # Use a specific project by name npx @cognite/cli@latest apps deploy --interactive -d your-project-name # Skip the build step (if already built) npx @cognite/cli@latest apps deploy --interactive --skip-build ``` ### Draft and published versions `apps deploy` uploads your app as a `DRAFT` version. To publish and activate it, see [How a version reaches Custom apps](/cdf/flows/concepts/app-lifecycle#how-a-version-reaches-custom-apps). `--published` flag is used for legacy apps: | Scenario | Behavior | | - | - | | Legacy app, `--published` flag | Honored — deploys as `PUBLISHED` | | App Hosting app, `--published` flag | Warning printed, flag ignored — always deploys as `DRAFT`. Use `apps publish .` / `apps activate .` instead | For App Hosting apps, use `npx @cognite/cli@latest apps publish .` and `npx @cognite/cli@latest apps activate .` to control visibility. ## App Hosting lifecycle commands For apps using App Hosting infrastructure (`"infra": "appsApi"` in `app.json`), use the following commands to manage the version lifecycle after deployment. Each version progresses through the states: `DRAFT` → `PUBLISHED` → `ACTIVE`. ### Check status Print the current deployment status of your app version: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps status . ``` ### Publish a version Transition the current version from `DRAFT` to `PUBLISHED` (visible but not yet the `ACTIVE` version): ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps publish . ``` This command is idempotent — re-running on an already `PUBLISHED` or `ACTIVE` version is a no-op. Publishing **requires a developer signature**, including on projects with [Dev status](/cdf/flows/guides/dev-status). Production projects without Dev status need **both** that signature and Cognite's app certification signature (`.cert.sig`) from [application certification](/cdf/flows/guides/application-certification). See [Code signing](/cdf/flows/guides/code-signing). ### Activate a version Set the current version as `ACTIVE` (the version users see). Publish the version if it's still `DRAFT`: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps activate . ``` If a previous version was `ACTIVE`, the output reports which version it replaced. Use `--interactive` to authenticate via browser sign-in when running these commands from your machine: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps status . --interactive npx @cognite/cli@latest apps publish . --interactive npx @cognite/cli@latest apps activate . --interactive ``` ## Automated deployment (CI/CD) For CI/CD pipelines, configure client credentials. ### Prerequisites 1. **OAuth client** — A client ID registered in your CDF project. 2. **Client secret** — The secret for that client. 3. **Capabilities** — The `apphosting:read` and `apphosting:write` capabilities granted to the client via a group. These capabilities can be scoped to the app's `externalId` from `app.json`. Add `apphosting:run` if the same client also needs to run the app. See [Assign capabilities](/cdf/access/guides/capabilities#flows-custom-apps). Ask your Cognite administrator to register an OAuth client with the required permissions. ### Step 1: Configure app.json Add `deployClientId` and `deploySecretName` to your deployment configuration: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "name": "My App", "description": "My awesome Flows app", "externalId": "my-app", "versionTag": "1.0.0", "deployments": [ { "org": "my-org", "project": "my-project", "baseUrl": "https://api.cognitedata.com", "deployClientId": "your-client-id", "deploySecretName": "MY_APP_SECRET" } ] } ``` | Field | Description | | - | - | | `name` | Display name in CDF | | `externalId` | Unique identifier (kebab-case) | | `versionTag` | Semantic version | | `deployClientId` | OAuth client ID | | `deploySecretName` | Environment variable name for the secret | | `idpType` | Identity provider (`cdf` or `entra_id`). Defaults to `cdf` | | `tenantId` | Microsoft Entra tenant ID (required when `idpType` is `entra_id`) | | `published` | Legacy field: whether the app is visible to end users. Prefer `apps publish` / `apps activate` for App Hosting apps. | #### Microsoft Entra ID deployment Set `idpType` to `entra_id` and provide `tenantId`: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "deployments": [ { "org": "my-org", "project": "my-project", "baseUrl": "https://api.cognitedata.com", "deployClientId": "your-entra-client-id", "deploySecretName": "MY_APP_SECRET", "idpType": "entra_id", "tenantId": "00000000-0000-0000-0000-000000000000" } ] } ``` When `idpType` is `entra_id`, Flows requests a token from Microsoft Entra ID and derives the CDF scope from `baseUrl` (for example, `https://greenfield.cognitedata.com/.default`). ### Step 2: Set the secret Export the client secret: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} export MY_APP_SECRET="your-client-secret" ``` In CI/CD, store secrets in your platform’s secret manager (for example, GitHub Actions repository secrets, Azure DevOps variable groups, or GitLab CI variables). ### Step 3: Deploy ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run deploy ``` This will: 1. Build your app (`tsc && vite build`). 2. Package `dist/` into a zip. 3. Authenticate with client credentials. 4. Upload through the CDF Files API. 5. Make the app available in CDF. ### Preview deployments For pull request previews: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run deploy-preview ``` ### Multiple environments Add multiple entries to `deployments`: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "deployments": [ { "org": "your-org", "project": "dev-project", "baseUrl": "https://api.cognitedata.com", "deployClientId": "dev-client-id", "deploySecretName": "DEV_SECRET" }, { "org": "your-org", "project": "prod-project", "baseUrl": "https://api.cognitedata.com", "deployClientId": "prod-client-id", "deploySecretName": "PROD_SECRET" } ] } ``` ## Versioning Update `versionTag` in `app.json` for each release: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "versionTag": "1.2.0" } ``` ### Roll back to a published version Each version is stored in CDF. To point **Custom apps** at an earlier published, Cognite-certified version, set `versionTag` in `app.json` to that version, then run: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps activate . ``` Do not run `apps deploy` against that `versionTag`. Published versions cannot be overwritten. See [New versions and rollback](/cdf/flows/concepts/app-lifecycle#new-versions-and-rollback) for when to roll back versus recertify. ## Troubleshooting ### What permissions do I need to deploy? Your OAuth client (for CI/CD) or user account (for interactive deployment) needs `apphosting:read` and `apphosting:write`. You can scope these capabilities to the app's `externalId` from `app.json`. Add `apphosting:run` if the same principal also needs to run the app. See [Assign capabilities](/cdf/access/guides/capabilities#flows-custom-apps). ### "Deployment secret not found" Ensure the environment variable name matches `deploySecretName`: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} export MY_APP_SECRET="your-secret" ``` ### "Failed to authenticate" Check that: * The client ID is correct. * The client secret is valid and not expired. * The client has the required CDF permissions (including Files API write access). ### Build fails Run the build locally: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run build ``` ### Interactive sign-in issues 1. Confirm you have access to the CDF project. 2. Ensure pop-ups are not blocked. 3. Try `npx @cognite/cli@latest apps deploy --interactive --org your-org` as a hint. ## Further reading * [Get started with Flows](/cdf/flows/guides/getting-started) — Create an app and first deploy. * [Run your app locally](/cdf/flows/guides/running-locally) — Dev server, HTTPS, and troubleshooting. * [App lifecycle](/cdf/flows/concepts/app-lifecycle) — Draft, published, and active versions. * [Code signing](/cdf/flows/guides/code-signing) — Sign bundles with a developer signature. * [Dev status](/cdf/flows/guides/dev-status) — Skip application certification on non-production projects. * [Auth API](/cdf/flows/reference/api/auth) — Tokens and SDK usage related to deployment identities. # Requesting Dev status for a CDF project Source: https://docs.cognite.com/cdf/flows/guides/dev-status Learn how to request Dev status for a non-production Cognite Data Fusion (CDF) project so you can publish builder-signed custom apps without application certification. Use this procedure to share a prototype application on a **test, dev, or staging** Cognite Data Fusion (CDF) project without [application certification](/cdf/flows/guides/application-certification). Dev status moves app security responsibility to the certified builder and the CDF project owner. Dev status transfers risk ownership for app security to the **certified builder** and the **CDF project owner**. Cognite does not run application certification on those applications. The builder and project owner own the checks Cognite normally performs, including the `/flows-code-review` skill, dependency / CVE follow-up (`review-packages.md` / `npm audit`), and not shipping known-bad or malicious code. Do not request Dev status for a production project. ## Prerequisites * Only a **CDF project owner** can submit this request. If you are not the owner, ask them to submit it. * The target project is **test, dev, or staging**. Cognite does not grant Dev status for production projects. * [Builder certification](/cdf/flows/guides/builder-certification) and a developer signature are required to publish after Cognite grants Dev status. See [Code signing](/cdf/flows/guides/code-signing). On a project with Dev status: * You still add a **developer signature** with `apps sign`. See [Code signing](/cdf/flows/guides/code-signing). * You can publish a builder-signed app. Cognite's **app certification signature** (`.cert.sig`) is not required, and CDF does not show the 14-day production warning. * Only the version owner can open an unsigned version. A builder-signed draft is shareable. See [App lifecycle](/cdf/flows/concepts/app-lifecycle). ## Request Dev status [Zendesk](https://cognite.zendesk.com) is Cognite's support-request site. Select the **Request Dev status for a CDF project** card to open a pre-filled form. Edit the form if needed, and then select **Submit**. If you can't sign in, ask your Cognite representative for a Zendesk login. Opens a pre-filled Zendesk request. ## After Cognite grants Dev status Follow [Code signing](/cdf/flows/guides/code-signing) for the developer signature. Follow [Deploying Flows custom apps](/cdf/flows/guides/deploying) and [App lifecycle](/cdf/flows/concepts/app-lifecycle). **Publish** locks the version. **Activate** makes the app show up on **Custom apps**, or replaces the previously active version. ## Further reading * [Application certification](/cdf/flows/guides/application-certification) — The default path you are opting out of. * [Builder certification](/cdf/flows/guides/builder-certification) — Qualify to build Flows custom apps in customer environments. * [Code signing](/cdf/flows/guides/code-signing) — Sign bundles with a developer signature. * [Deploying Flows custom apps](/cdf/flows/guides/deploying) — Interactive and CI/CD deployment, lifecycle commands, and rollback. # Getting started with Flows custom apps Source: https://docs.cognite.com/cdf/flows/guides/getting-started Create a new Flows custom app, install dependencies, open it in Cursor, and run it inside Cognite Data Fusion (CDF). **Flows** (`@cognite/cli`) scaffolds React applications for CDF so you can focus on product features instead of wiring authentication and build tooling. The generated stack uses **TypeScript**, **Vite**, the Cognite SDK through React hooks, and paths to deploy to CDF. ## Prerequisites Before you begin, ensure you have the following: * **Latest supported Node.js and npm versions.** The npm version that is installed with Node.js is sufficient. * **Git** — used by the Cognite CLI (`@cognite/cli`) to initialize and manage your app repository. * A coding IDE such as Cursor or VS Code. * A compatible coding agent. See [Supported AI development environments](/cdf/flows/concepts/skills#supported-ai-development-environments) for the full list. * Access to a **CDF** project (organization, project name, and cluster). * **App hosting capabilities** assigned by your administrator in CDF groups. See [Assign capabilities](/cdf/access/guides/capabilities#flows-custom-apps). * **Access to `cogniteappsdata.com`** in your network allowlist. If your organization uses a firewall or proxy that restricts domains, ensure this domain is allowlisted. For other URLs CDF may require, see [Allowlist URLs](/cdf/admin/allowlist). * A [trusted local HTTPS certificate](/cdf/flows/guides/local-https) as CDF won't load your app unless the browser trusts the dev server's certificate. Options exist with or without admin rights. **Certification for customer environments**: To publish to a **production** project, you need both a developer signature and Cognite's app certification signature. On a **non-production** project, a [certified builder](/cdf/flows/guides/builder-certification) can publish with a developer signature alone if the project has Dev status. Cognite does not grant Dev status for production projects. [Request Dev status](/cdf/flows/guides/dev-status) before you publish to a non-production project. See [Code signing](/cdf/flows/guides/code-signing). ## Set up your environment ### Install Node.js It's recommended to use **nvm** (Node Version Manager) to install and manage Node.js versions. ```bash nvm (recommended) wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Download and install nvm: curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash # Load nvm (or restart your terminal): . "$HOME/.nvm/nvm.sh" # Install a currently supported Node.js release: nvm install --lts ``` ```bash macOS (Homebrew) theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} brew install node ``` ```bash Windows theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Download the installer from https://nodejs.org/ ``` #### Verify installation ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node --version npm --version ``` If you need to upgrade your **Node.js** or **npm** versions: Use nvm or Homebrew. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} nvm install --lts nvm use --lts # OR if using Homebrew: brew upgrade node ``` Run the following command from your home directory `~` or outside your project folder to avoid local configuration conflicts. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm install -g npm@latest ``` ### Choose your AI development environment Flows apps work with any modern AI coding agent. Install your preferred environment before creating your app. For the full list of supported environments, see [Supported AI development environments](/cdf/flows/concepts/skills#supported-ai-development-environments). | Environment | Install | | - | - | | **Cursor** (recommended) | [cursor.com](https://cursor.com/) | | **Claude Code** | `npm install -g @anthropic-ai/claude-code` | | **GitHub Copilot** | [VS Code extension](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) | | **Windsurf** | [codeium.com/windsurf](https://codeium.com/windsurf) | | **Any other agent** | See [supported environments](/cdf/flows/concepts/skills#supported-ai-development-environments) | All environments listed above detect skills without manual setup. Open the generated project folder as the workspace root and your agent will find the skill files automatically. ### GitHub (optional) For version control, create an account at [github.com](https://github.com/) and configure Git: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} git config --global user.name "Your Name" git config --global user.email "your.email@example.com" ``` ### Cognite Data Fusion (CDF) access You need: * Your **organization** name (for example, `your-org-name`). * Your **project** name (for example, `your-project-name`). * Your **cluster** (for example, `your-cluster-name`). Contact your Cognite administrator if you need access. ## Create and run your app Run: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps create ``` Or create the app in a specific directory: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps create [directory] ``` Or create in the current directory by using a special `.` syntax: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps create . ``` This command is the same on Linux, macOS, and Windows. If `npx` prompts you to install the package or doesn't run, use `npm exec --yes -- @cognite/cli@latest apps create`. After a global install (`npm install -g @cognite/cli@latest`), you can also run `cognite apps create`. You are prompted for information such as app name (pre-filled if you provided a directory), display name, description, org, project, cluster, base URL, and whether to enable spec-driven development. | Prompt | Description | Example | | - | - | - | | **App name** | Kebab-case identifier. | `my-analytics-app` | | **Display name** | Human-readable name. | `My Analytics App` | | **Description** | Brief description. | `Analytics dashboard` | | **Org** | Your Cognite organization. | `your-org-name` | | **Project** | Your CDF project. | `your-project-name` | | **Cluster** | Cluster subdomain. | `your-cluster-name` | | **Base URL** | CDF cluster base URL, pre-filled from your cluster but can be changed. For private-link setups, replace with your internal URL. | `https://your-cluster-name.cognitedata.com` | | **Enable spec-driven development** | Adds a `SPEC.md` file and spec-kit slash commands to your app for structured feature planning. See [Spec-driven development](/cdf/flows/concepts/spec-driven-development). | `yes` or `no`. | Pass flags to skip individual prompts. This is useful in scripts or when you want to avoid interactive mode: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps create my-app \ --display-name "My App" \ --description "Analytics dashboard" \ --org your-org-name \ --project your-project-name \ --cluster your-cluster-name \ --base-url https://your-cluster-name.cognitedata.com \ --no-spec-kit ``` Any flag you omit will still be prompted interactively. If you plan to publish this app to a **production** environment, run `/flows-app-brief` before you start building. The skill coaches you through persona, problem, and success criteria, then writes `App-Brief.md` in the repository root. [Application certification](/cdf/flows/guides/application-certification) requires this file. Skip this step for projects with [Dev status](/cdf/flows/guides/dev-status). Cognite's app certification signature isn't required on those projects, but you still need a developer signature before you publish. See [Code signing](/cdf/flows/guides/code-signing). Open the generated folder in your preferred editor or coding agent environment. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cursor my-analytics-app ``` ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} claude my-analytics-app ``` You can run Claude Code in the terminal only, or through VS Code or any IDE with the Claude plugin installed. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} code my-analytics-app ``` Ensure the app folder is the workspace **root**. Your coding agent (Cursor, Claude Code, Copilot, etc.) detects rules, MCP configuration, and skills from the project root. Opening a parent directory or subfolder causes the agent to miss the skill files. Install dependencies and start the development server: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm install npm start ``` Check `engines` in the generated `package.json` for the versions the current template requires. The dev server serves your app over HTTPS using a self-signed certificate. On the first run, the browser will show a warning and CDF will display **Failed to connect to Fusion host** until you trust the certificate. See [Trust the local HTTPS certificate](/cdf/flows/guides/local-https) for the one-time `mkcert` setup, or the no-admin manual fallback. Your app opens in CDF when the dev server is ready. Use Cursor to initialize a Git repository and make your first commit. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} git init git add . git commit -m "Initial commit" ``` You can also use the Source Control panel in Cursor. Start building your application. See the following resources and tips for help: * [Run your app locally](/cdf/flows/guides/running-locally) — See the UI and behavior while you develop. * [Installing Aura](/cdf/aura/guides/installation) — UI framework that ships with the template. * [Flows features](/cdf/flows/concepts/features) — For a concise overview of everything the template gives you. * [Deploy your Flows app](/cdf/flows/guides/deploying) — See interactive and CI/CD options. When you start building: * **[Aura](/cdf/aura/guides/installation)** — Ensure **[`@cognite/aura`](/cdf/aura/guides/installation)** is listed under `dependencies` in `package.json` (for example `"@cognite/aura": "0.1.1"`). If it is missing, ask your coding agent to add that dependency and install it. * **`SPEC.md`** — Keep product intent in the app root so you and the AI share the same requirements as you iterate. * **SDK and data** — Call CDF through **`connectToHostApp()`** from **`@cognite/app-sdk`**; **TanStack Query** is already set up for loading and error handling around SDK calls. When you're ready to deploy your app to CDF, run: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps deploy --interactive ``` See [Deploy](/cdf/flows/guides/deploying) for options and CI/CD. ## Generated structure Here is the generated structure of your app: ## What the template includes The tree above shows where files live. When you run `npx @cognite/cli@latest apps create`, the generator also wires in the following **capabilities** (not only folders and config files): * **Project scaffolding** — Application layout, `app.json`, configuration files, and **skills** for AI-assisted development. * **Aura UI** — The template includes [`@cognite/aura`](/cdf/aura/guides/installation). See the [Aura design system](/cdf/aura) for foundations, primitives, and patterns. * **CDF authentication** — Use `connectToHostApp()` from `@cognite/app-sdk` to get an authenticated Cognite SDK inside CDF (see [Auth API](/cdf/flows/reference/api/auth)). * **Local development** — Vite with hot module replacement and HTTPS; run commands with **npm**. See [Run your app locally](/cdf/flows/guides/running-locally). * **Data fetching** — **TanStack Query (React Query)** is included to use with `connectToHostApp()` for SDK calls. * **Tests and quality** — **Vitest** and **React Testing Library**, plus linting/formatting via your template's scripts in `package.json`. * **Deployment** — Interactive deploy from your machine or automation in CI/CD. See [Deploy your Flows app](/cdf/flows/guides/deploying). * **Manifest and CSP** — `manifest.json` declares your app's metadata and `permissions.network` for allowed origins under a strict Content Security Policy. See [Content Security Policy](/cdf/flows/reference/api/csp). For more detail on each area (including TypeScript, examples, and build commands), see [Flows features](/cdf/flows/concepts/features). ## Further reading * [Flows features](/cdf/flows/concepts/features) — Scaffolding, auth, Vite, tests, deploy, and data fetching at a glance. * [Aura design system](/cdf/aura) — Foundations, primitives, and patterns for Flows custom app UI. * [Run your app locally](/cdf/flows/guides/running-locally) — Dev server, HTTPS, CDF iframe, and configuration. * [Deploy your Flows app](/cdf/flows/guides/deploying) — Interactive and automated deployment. * [Builder certification](/cdf/flows/guides/builder-certification) — Required to build Flows custom apps in customer environments. * [Application certification](/cdf/flows/guides/application-certification) — Required per app before production deployment. * [Dev status](/cdf/flows/guides/dev-status) — Skip application certification on non-production projects. * [Auth API](/cdf/flows/reference/api/auth) — `connectToHostApp` and authenticated CDF access. * [Host App API](/cdf/flows/reference/api/host-app-api) — Navigation, state, sidebar visibility, fullscreen, and agent integration. * [Content Security Policy](/cdf/flows/reference/api/csp) — Production CSP baseline and how to extend it via `manifest.json`. # Developing and shipping Flows custom apps Source: https://docs.cognite.com/cdf/flows/guides/index Follow Flows custom app guides from first project creation through local development with Vite to deploying React apps inside Cognite Data Fusion (CDF). Walk through creating a **Flows custom app**, running it locally, and publishing it to CDF in a single guided path. Before you build for customers, review the [quality guidelines](/cdf/flows/guides/quality-guidelines) — they define the standards your app must meet for certification. Create an app, install dependencies, and run it in CDF. Run specify, clarify, and plan on a real example and end with a reviewable implementation plan. Persist and share app state through URLs with `syncInternalState`. Dev server, HTTPS, CDF iframe, and local configuration. One-time `mkcert` setup, plus a no-admin fallback. Use the App Hosting OAuth relay for an additional Microsoft Entra ID sign-in from within a Flows custom app. Interactive deploy and CI/CD. Engineering and design standards every Flows app must meet. Generate signing keys and sign app bundles from the CLI. # Trusting the local HTTPS certificate for Flows custom apps Source: https://docs.cognite.com/cdf/flows/guides/local-https Step-by-step guide to installing a trusted local certificate so your Flows custom app loads without warnings in Cognite Data Fusion (CDF). This guide is for developers building Flows custom apps. Before you begin, make sure you have a Flows custom app created and dependencies installed. See [Getting started with Flows custom apps](/cdf/flows/guides/getting-started). `npm run dev` serves your app over HTTPS so CDF can embed it in an iframe. Without a trusted certificate, the browser blocks the embed and shows **Not Secure** or **"the webpage at `https://localhost:3001/` might be having issues"**. You need to establish trust between your browser and the local dev server before CDF can load your app. You can establish trust in two ways: * **[Install `mkcert`](#installing-mkcert-recommended)**: One-time setup that works everywhere. Requires admin access. * **[Accept the certificate in the browser](#accepting-the-certificate-manually-no-admin-required)**: No installation needed, but you must repeat this on every dev server restart. ## Installing mkcert (recommended) Run the following command in your terminal: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} brew install mkcert ``` 1. Open the **Start** menu, type **Windows PowerShell**. 2. Right-click **Windows PowerShell** and select **Run as administrator**. 3. Accept the UAC prompt. 4. Run the following command: ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} choco install mkcert ``` Run the following command in your terminal: ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} scoop bucket add extras scoop install mkcert ``` If you don't have a package manager, download the binary directly from the [mkcert installation guide](https://github.com/FiloSottile/mkcert#installation). ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps setup-https ``` This command adds the `mkcert` **Certificate Authority (CA)** to your system trust store and issues a certificate for `localhost`, `local.cognite.ai`, and `*.local.cognite.ai`. You must accept the sudo or UAC prompt when it appears. Stop `npm run dev`, restart it, then fully quit and reopen your browser so it re-reads the trust store. The address bar should show the lock icon. ## Accepting the certificate manually (no admin required) If you don't have admin access, manually trust the dev server's self-signed certificate in your browser: Run the following command in your terminal: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run dev ``` The URL is displayed in the terminal (for example, `https://localhost:3001`). Open a new tab in your browser and paste the URL. The browser shows a privacy warning: Select **Advanced** → **Proceed to localhost (unsafe)**. Select **Advanced…** → **Accept the Risk and Continue**. Select **Show Details** → **visit this website** → **Visit Website**. The new tab will appear blank or show a "The webpage at `https://localhost:3001` might be having issues" message as the app only renders within the CDF iframe, not in a standalone window. You can close this tab. Return to the **CDF** tab and reload. If the iframe doesn't load immediately, reload again and wait a few seconds for the now-trusted session to register. Install `mkcert` to avoid repeating this step.Without `mkcert`, the dev server generates a new certificate on every start, so the browser warning returns on every restart, port change (for example, `3002` instead of `3001`), or browser switch. ## Troubleshooting **Symptom**: * Running `npx @cognite/cli@latest apps setup-https` fails with `mkcert is not installed`. **Cause**: * The CLI cannot find `mkcert` on your `PATH`. **Resolution**: * Install `mkcert` (see [Installing mkcert (recommended)](#installing-mkcert-recommended)), then run the command again. * On Windows, run `choco install mkcert` from an elevated PowerShell. **Symptom**: * The browser shows **Not Secure** even after `apps setup-https` completed successfully. **Cause**: * The browser cached the old trust store state and hasn't re-read it yet. **Resolution**: * Fully quit and reopen the browser. * On Windows, close Edge from the system tray, not just the window. **Symptom**: * The CLI reports that the `apps setup-https` command doesn't exist. **Cause**: * Your local version of `@cognite/cli` is older than `1.4.0`, which is when the command was added. **Resolution**: * Run the command with `@latest` to bypass any pinned version: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npx @cognite/cli@latest apps setup-https ``` * If `@latest` still resolves to a version older than `1.4.0`, delete any `.npmrc` in the current directory that pins a registry or version. **Symptom**: * Each `npm run dev` start uses a higher port number. **Cause**: * An orphaned `node` process from a previous session is still holding the port. **Resolution**: * Find and stop the process. You can only stop processes you own. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # macOS / Linux lsof -i :3001 # Checks what's running on port 3001 kill -9 # Stops the process ``` ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Windows Get-Process -Id (Get-NetTCPConnection -LocalPort 3001).OwningProcess Stop-Process -Id -Force ``` ## Further reading * [Running Flows custom apps locally](/cdf/flows/guides/running-locally) * [Sign Flows custom apps from the CLI](/cdf/flows/guides/code-signing) # Signing in with an additional Microsoft Entra ID account Source: https://docs.cognite.com/cdf/flows/guides/oauth-relay Step-by-step guide to using the App Hosting OAuth relay for an additional Microsoft Entra ID sign-in from within a Flows custom app. Use the App Hosting OAuth relay when your Flows custom app needs an additional sign-in from within the app. For example, pull data from a source authenticated by a different Microsoft Entra ID tenant than Cognite Data Fusion (CDF). [`connectToHostApp`](/cdf/flows/reference/api/auth) returns CDF credentials only, so the user must sign in with a different Microsoft Entra ID account, typically in that other tenant. You can run the flow more than once for multiple additional accounts. You implement the popup in your app code and decide when it runs. App Hosting serves the relay as a static, unauthenticated page at `/apphosting/oauth-relay` on the app's origin; the path stays the same across versions. The authorization code stays in the browser: it isn't sent to CDF or App Hosting, isn't persisted beyond the browser cache, and may be accessible across sessions. ## Prerequisites * A [Flows custom app](/cdf/flows/guides/getting-started) that runs inside CDF. * A Microsoft Entra ID app registration for the other tenant or service, with the [single-page application](#register-the-redirect-uri) platform. * CDF credentials from [`connectToHostApp`](/cdf/flows/reference/api/auth). Use the relay only for the additional Microsoft Entra ID sign-in, not for CDF. ## How the relay works 1. Your app opens a popup at the Microsoft Entra ID authorize URL with `response_mode=fragment`, so Microsoft Entra ID returns the authorization code in the URL fragment (`#code=...`). 2. Include `prompt=select_account` on that authorization request so Microsoft Entra ID shows the account picker. Without it, Microsoft Entra ID may reuse the CDF account, and the app won't get a token for the other tenant. 3. After sign-in, Microsoft Entra ID redirects the popup to `{app-origin}/apphosting/oauth-relay#code=...&state=...`. 4. The relay posts the fragment to `window.opener` and does not store it: ```js theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { type: "APPHOSTING_OAUTH_RESPONSE", fragment: window.location.hash } ``` The relay uses `postMessage` to send the fragment to the window that opened the popup (`window.opener`). The `postMessage` target origin is the relay's own origin. If there is no opener, the page does nothing. 5. Your app checks `event.origin`, reads `code` from the fragment, and exchanges it for tokens with PKCE (Proof Key for Code Exchange). You store those tokens and attach them to calls to the other service. See [Store the token and call the other service](#store-the-token-and-call-the-other-service). The relay reads `window.location.hash`. If you omit `response_mode=fragment`, the code lands in the query string and the app receives an empty fragment. ## Find the deployed app origin You need the app origin to register the redirect URI in Microsoft Entra ID. The origin stays the same for every version of the app in a project. For example, the origin is `https://9f446ebf4f46.apps.api.cogniteappsdata.com`. You can copy it from the browser or compute it from the app's `externalId` and `project`. Open the deployed app in CDF and copy the origin from the address bar. Keep `https://` and the host, and drop everything after the first `/` that follows the host. Assemble `https://.apps..cogniteappsdata.com` from `app.json`. `{cluster}` is the `baseUrl` subdomain (`api` for `https://api.cognitedata.com`). The command prints `{app-hash}`: the first 12 characters of the SHA-256 hash of `externalId`, a null byte, and `project`. Replace `my-app` and `my-project` with your `externalId` and `project`. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} printf 'my-app\0my-project' | shasum -a 256 | cut -c1-12 ``` ```powershell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} $sha = [System.Security.Cryptography.SHA256]::Create() $bytes = [System.Text.Encoding]::UTF8.GetBytes("my-app" + [char]0 + "my-project") (-join ($sha.ComputeHash($bytes) | ForEach-Object { $_.ToString("x2") })).Substring(0, 12) ``` Use that origin when registering the redirect URI. The redirect URI is the origin plus `/apphosting/oauth-relay`. ## Register the redirect URI In the Azure portal, add the redirect URIs to the **Single-page application** platform. Don't use the **Web** platform because it doesn't return the CORS headers required to exchange the code from the iframe. Go to **Microsoft Entra ID** → **App registrations** → your registration → **Authentication**. Under the **Redirect URI configuration** tab, select **+ Add a platform** or **+ Add Redirect URI**. In the right side panel, select **Single-page application**. Add the redirect URI for local and one for the deployed app. The path is always `/apphosting/oauth-relay`. | Environment | Redirect URI | | - | - | | Local | `https://localhost:3001/apphosting/oauth-relay` | | Deployed | `https://.apps..cogniteappsdata.com/apphosting/oauth-relay` | Don't register an app asset such as `callback.html` on the deployed origin. App assets require the CDF session cookie, so the popup receives a 401 response and the opener can't read it. After you add the URI, select **Confirm** to update authentication and save the changes. ## Serve the relay locally App Hosting serves `/apphosting/oauth-relay` in production. For local development, serve the same path from the Vite dev server: ```ts wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import type { PluginOption } from "vite"; function oauthRelayDevPlugin(): PluginOption { const relayHtml = ` OAuth relay

    Completing sign-in…

    `; return { name: "oauth-relay-dev", configureServer(server) { server.middlewares.use("/apphosting/oauth-relay", (_req, res) => { res.setHeader("Content-Type", "text/html"); res.end(relayHtml); }); }, }; } ``` Add `oauthRelayDevPlugin()` to the `plugins` array in `vite.config.ts`. ## Handle the relay response The relay posts the fragment to your app with `postMessage`. Handle the `message` event and read the authorization `code` from the fragment. Then POST the code and the PKCE verifier to the Microsoft Entra ID token endpoint to get tokens. ```ts wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} window.addEventListener("message", (event) => { if (event.origin !== window.location.origin) return; if (event.data?.type !== "APPHOSTING_OAUTH_RESPONSE") return; const params = new URLSearchParams( String(event.data.fragment).replace(/^#/, "") ); const code = params.get("code"); // Next: POST code + PKCE verifier to the Microsoft Entra ID token endpoint. }); ``` Allow the identity provider in `manifest.json`. If you call Microsoft Graph (Microsoft's API for user and organization data, for example the `/me` endpoint), also allow `https://graph.microsoft.com`. See [Content Security Policy](/cdf/flows/reference/api/csp). ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": [ "https://login.microsoftonline.com", "https://graph.microsoft.com" ], "directives": ["connect-src"] } ] } } ``` ## Store the token and call the other service The relay doesn't store the authorization code or the tokens. After you exchange the code, keep the access token (and refresh token, if returned) in your app, for example, in memory or in `localStorage`. Attach the access token to the requests you send to the other service. ## Refresh the access token If the token response includes a `refresh_token`, redeem it before the access token expires. Microsoft Entra ID single-page application (SPA) registrations rotate refresh tokens, so replace the stored refresh token after each successful response. ## Further reading * [Flows Auth API](/cdf/flows/reference/api/auth): Get a CDF token with `connectToHostApp`. * [Content Security Policy](/cdf/flows/reference/api/csp): Allow `login.microsoftonline.com` in `manifest.json`. * [Run your app locally](/cdf/flows/guides/running-locally): Use the development server and CDF iframe. # Flows custom apps quality guidelines Source: https://docs.cognite.com/cdf/flows/guides/quality-guidelines Comprehensive quality assessment criteria for Flows custom apps, including design evaluation rubric and engineering standards. These guidelines define the quality standards that Flows custom apps must meet for certification. They're the rubric the design review step of the [Application certification](/cdf/flows/guides/application-certification) flow scores against. If you ran `/flows-design-review`, the skill already proposed scores against this rubric and wrote them to `reviews/design-review/feedback-round-N/design-review-report.md`. Use this page as the authoritative reference when overriding any of the proposed scores or interpreting reviewer feedback. ## Design quality assessment The design quality assessment evaluates user experience, interface design, and usability through 10 key questions based on established UX principles. **Target score: 3.8 or higher average** across all questions. ## Self-assessment process ### Step 1: Confirm the user and tasks **Before opening the app**: 1. **Define the primary user**: Identify the specific role and context (e.g., "Maintenance engineers on offshore platforms") 2. **List key tasks**: Document 2-3 critical workflows this user needs to complete (e.g., "Check pump vibration alerts", "Schedule maintenance work") 3. **Gather context**: Note the user's experience level, time constraints, and success criteria This step is crucial. The assessment is only as useful as the clarity of the user and tasks. If they aren't well-defined, stop here and define them first. ### Step 2: Walk through each task end-to-end **Task-based evaluation**: 1. **Open the app as that user**: Use a clean browser session with representative test data 2. **Complete each task from beginning to end**: Click through every step without shortcuts 3. **Focus on task completion**: Don't evaluate screens in isolation - only findings that affect task completion matter 4. **Document pain points**: Note where users get stuck, confused, or make errors ### Step 3: Score the 10 questions **Assessment process**: 1. **Apply the scoring criteria**: Use what you observed during task completion to score each question 1-5 2. **Focus on critical areas**: Questions 2 (navigation) and 6 (error prevention) require walking the tasks and can't be assessed from code alone 3. **Calculate your average**: Sum all scores ÷ 10 4. **Identify improvement areas**: List any questions scoring under 3 for immediate attention ## The 10 design quality questions **Question**: Are you using [Aura](/cdf/aura) tokens, layouts, components and patterns correctly? See the [Aura design system](/cdf/aura) for foundations, primitives, and patterns. **5 - Excellent**: All Aura tokens are applied correctly, no hard coded values are used. The app uses proper responsive sizing and page layouts. Aura components are used without style overrides. Component best practices followed. Exceptions allowed for component needs not found in the system. Patterns align with design system guidelines. **4 - Good**: Mostly using Aura tokens and components with 1-2 minor exceptions. Layout spacing mostly consistent. Minimal style overrides that don't break patterns. **3 - Average**: Mix of Aura and custom elements. Some proper spacing, some random values. Using Aura components but overriding styles in multiple places. Some best practices ignored. **2 - Below average**: Frequently using custom colors, typography, or spacing instead of Aura tokens. Heavy component customization that breaks design system patterns. **1 - Poor**: Not using Aura design system. Custom colors, fonts, spacing throughout. Building custom components instead of using Aura. No adherence to design system guidelines. **Question**: Can users tell where they are and navigate easily? Are elements organized with clear visual hierarchy? **5 - Excellent**: User's current location always clear. Easy navigation forward/back. Menu structure consistent. Related items clearly grouped. Strong visual hierarchy guides attention. Content flows logically (F/Z pattern). **4 - Good**: Usually clear where you are. Navigation mostly consistent. Good grouping and hierarchy with minor exceptions. **3 - Average**: Sometimes unclear what page you're on. Navigation works but isn't always intuitive. Some grouping makes sense, hierarchy exists but not always clear. **2 - Below average**: Often lost or confused. Navigation changes between pages. Poor grouping and weak visual hierarchy - everything looks equally important. **1 - Poor**: No indication of current location. No clear way to navigate. Inconsistent structure. No logical grouping or visual hierarchy. **Question**: Are buttons, inputs, and actions labeled clearly? Can users understand what everything does? **5 - Excellent**: Every element has a clear, specific label. Language is plain and action-oriented ("Save changes", "Delete item"). **4 - Good**: Most labels are clear with only minor ambiguity. Language is mostly plain with minimal jargon. **3 - Average**: Labels are present but sometimes vague ("Submit", "OK"). Some unnecessary jargon used. **2 - Below average**: Many labels are unclear or generic. Heavy use of technical terms without explanation. **1 - Poor**: Labels are missing, confusing, or use jargon that users won't understand. **Question**: Do users know what's happening? Are forms easy to use with helpful validation and error messages? **5 - Excellent**: Immediate feedback for actions. Clear loading states. Helpful success/error messages that explain what happened and what to do next. All form fields are clearly labeled. Required fields marked. Real-time validation with specific, actionable messages. Easy to correct mistakes. **4 - Good**: Most actions provide feedback. Loading states present. Form fields well-labeled. Validation is mostly helpful with minor timing or clarity issues. Error messages are generally clear. **3 - Average**: Some feedback provided but not consistent. Loading states sometimes missing. Fields labeled but some unclear. Basic validation present. Error messages are generic and could be more helpful. **2 - Below average**: Minimal feedback. Users often don't know if actions worked. Unclear what fields need. Validation only happens on submit. Unhelpful error messages. **1 - Poor**: No feedback. Silent failures. Technical error codes without explanation. No labels or requirements shown. No validation or only shows errors after multiple attempts. **Question**: Is it obvious what's clickable? Do interactive elements respond appropriately? **5 - Excellent**: All clickable items look clickable (buttons look like buttons). Hover effects on interactive elements. Cursor changes appropriately. **4 - Good**: Most interactive elements are obvious. Hover effects mostly present. Minor inconsistencies. **3 - Average**: Some clickable items look clickable. Inconsistent hover states. Occasionally unclear what's interactive. **2 - Below average**: Many interactive elements don't look clickable. Few or no hover effects. Confusing interactions. **1 - Poor**: Can't tell what's clickable. No visual feedback on interaction. Elements that look clickable aren't. **Question**: Can users undo or cancel destructive actions? Are there warnings before important actions? **5 - Excellent**: Confirmation dialogs before destructive actions. Auto-save prevents data loss. Clear undo options or cancel buttons. **4 - Good**: Most destructive actions have warnings. Some auto-save or undo capabilities. **3 - Average**: Some warnings present for major actions. Limited undo/cancel options. Some risk of data loss. **2 - Below average**: Few warnings. No undo options. Easy to accidentally lose work. **1 - Poor**: No warnings. No way to undo or cancel. Frequent accidental data loss. **Question**: Does your app work well on different screen sizes? Is it usable on desktop, tablet, and mobile? **5 - Excellent**: Seamless experience across desktop, tablet, and mobile. Layouts adapt appropriately. Touch targets are 40px+ on mobile. Text readable on all screen sizes. Navigation optimized for each device. No horizontal scrolling. If app is not used on mobile/hand-held device, it should be hidden, or functionality should be limited to prevent a poor user experience. Are hover states accounted for in mobile/touch devices so users can still access the same content without the ability to hover. Smooth behavior allows for relevant content to remain visible on mobile devices when native elements, such as keyboards, are present. **4 - Good**: Works well on most devices. Layouts adapt with minor issues. Most touch targets are appropriate. Generally readable and usable across screen sizes. **3 - Average**: Functional on multiple devices but not optimized. Some layout issues on smaller screens. Some touch targets too small on mobile. Horizontal scrolling sometimes required. **2 - Below average**: Poor mobile/tablet experience. Layouts break on smaller screens. Many touch targets too small. Text hard to read on mobile. Significant usability issues on non-desktop devices. **1 - Poor**: Only works on desktop. Completely broken on mobile/tablet. Unusable on smaller screens. **Question**: When there's no data, is it clear what to do next? Do empty pages guide users? Can users easily request access or share data (invite) with other users? **5 - Excellent**: All empty states show helpful messages and clear next steps. Simple graphics guide users. First-time users know exactly what to do. **4 - Good**: Most empty states are helpful. Generally clear what to do first. Minor gaps in guidance. **3 - Average**: Some empty states explained. First-time users can figure it out with effort. **2 - Below average**: Many blank pages with no guidance. Unclear how to get started. **1 - Poor**: Blank pages everywhere. No indication of what to do or why nothing is showing. **Question**: Does the app load quickly? Can users complete tasks efficiently? Are there shortcuts for common actions? **5 - Excellent**: Fast loading with progressive content display. Bulk actions available. Keyboard shortcuts work. Common tasks take minimal clicks. **4 - Good**: Reasonable loading times. Most tasks are streamlined. Some efficiency features present. **3 - Average**: Acceptable performance but could be faster. Tasks require moderate effort. Few shortcuts. **2 - Below average**: Slow loading. Tasks require many steps or clicks. No shortcuts or bulk actions. **1 - Poor**: Very slow or unresponsive. Tasks are tedious and require excessive clicks. No efficiency features. **Question**: Can people use your app with technology assisted devices, such as screen readers and keyboard inputs? Are there focus states and do they navigate across the page correctly? Does it meet WCAG AA 2.1 standards? **5 - Excellent**: All interactions work via keyboard (Tab, Enter, Escape). Text contrast meets WCAG AA (4.5:1 for normal text, 3:1 for large text). Clear focus indicators on all interactive elements. Proper ARIA labels for screen readers. Images have alt text. Clickable areas are 40px+ for touch, 20px+ for mouse. Form errors announced to screen readers. **4 - Good**: Most accessibility requirements met. Keyboard navigation works for nearly all features. Contrast meets standards with 1-2 minor exceptions. Focus indicators mostly visible. Most ARIA labels present. **3 - Average**: Basic keyboard support but missing for some features. Contrast mostly acceptable but some failures. Focus indicators are present but not always clear. Some ARIA labels missing. Clickable areas sometimes too small. **2 - Below average**: Limited keyboard support. Multiple contrast failures. Weak or missing focus indicators. Few ARIA labels. Many clickable areas are too small. Screen reader experience is poor. **1 - Poor**: No keyboard navigation. Poor contrast throughout. No focus indicators. No ARIA labels. Not usable with assistive technologies. ## Quality scoring Each question is scored on a scale of 1-5, then calculate your average score by dividing the total by 10. | Average score | Quality level | Recommendation | | - | - | - | | **4.5 - 5.0** | **Excellent — ready to launch** | Your app meets high quality standards. Minor improvements can be made over time. | | **3.8 - 4.4** | **Good — launch with minor fixes** | Your app is functional and usable. Address the lower-scoring areas before or shortly after launch. | | **3.0 - 3.7** | **Average — needs improvement** | Your app works but has significant usability issues. Fix major problems before launching. | | **Below 3.0** | **Needs significant work** | Your app requires substantial improvements before launch. Focus on the lowest-scoring areas first. | ## Further reading * [Application certification](/cdf/flows/guides/application-certification) — Where this rubric is applied during per-app certification. * [Builder certification](/cdf/flows/guides/builder-certification) — Individual developer certification process. * [Skills](/cdf/flows/concepts/skills) — How the certification skills (including the design review skill) are delivered. * [Get started with Flows](/cdf/flows/guides/getting-started) — Initial setup and development guide. # Running Flows custom apps locally Source: https://docs.cognite.com/cdf/flows/guides/running-locally Run your Flows custom app in development mode with npm, the Vite dev server, HTTPS, and Cognite Data Fusion (CDF). Local development uses **Vite** (hot module replacement, HTTPS on localhost) and **npm**. The [Vite plugin API](/cdf/flows/reference/api/vite) documents `fusionOpenPlugin` and the CDF development URL pattern. ## Start the dev server After you create your Flows custom app and install dependencies, start the dev server to run the app locally: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm start # or npm run dev ``` This will: 1. Generate HTTPS certificates (first run only). 2. Start the Vite dev server on `https://localhost:3001` (or the next available port if 3001 is already in use). 3. Open your app in CDF (when configured). You access your app through the CDF URL shown in [Development URL](#development-url) below, not directly at `localhost:3001`. The browser needs to reach the local dev server, but authentication requires the CDF parent window. The dev server generates a self-signed HTTPS certificate to embed in CDF. The browser does not trust this certificate by default, so you'll see a warning, and CDF will show **Failed to connect to Fusion host** until you trust it. See [Trust the local HTTPS certificate](/cdf/flows/guides/local-https) for the one-time setup. ## How it works Users sign in through CDF. Your app runs in an iframe inside CDF, and `connectToHostApp()` from `@cognite/app-sdk` handles authentication by communicating with the host window. Use the returned `HostAppAPI` to get an access token and construct an authenticated Cognite SDK. **TanStack Query (React Query)** is included for data fetching. Combine it with `connectToHostApp()` to call the SDK (for example, assets or time series) with loading and error handling in React. See the [Auth API](/cdf/flows/reference/api/auth) for connection and credential usage. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} %%{init: { "theme": "base", "themeVariables": { "lineColor": "#b8bec6", "primaryColor": "#ffffff", "tertiaryColor": "#f6f8fa", "fontFamily": "Inter, system-ui, -apple-system, Segoe UI, Roboto, Ubuntu, Cantarell, Noto Sans, Helvetica Neue, Arial, sans-serif", "fontSize": "14px" } }}%% flowchart TB %% ===== Node styles (same palette as Records / streams diagrams) ===== classDef Site fill:#2f6cf6,stroke:#24292f,stroke-width:2px,color:#fff classDef Area fill:#8e43e7,stroke:#24292f,stroke-width:2px,color:#fff classDef Line fill:#0fa678,stroke:#24292f,stroke-width:2px,color:#fff classDef Record fill:#566370,stroke:#24292f,stroke-width:2px,color:#fff linkStyle default stroke:#b8bec6,stroke-width:3px; subgraph fusion["Cognite Data Fusion"] subgraph app["Your Flows app"] direction TB dev["Vite dev server on localhost:3001"]:::Line creds["Receives auth credentials from CDF"]:::Record dev --> creds end end class fusion Site; class app Area; ``` ## Development URL The dev server can open your app at: ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} https://{org}.fusion.cognite.com/{project}/flows-apps/development/{appExternalId}/3001 ``` Where `{org}` and `{project}` come from your `app.json` configuration. ## Hot module replacement Changes to your code appear quickly without losing state: ```tsx theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function App() { return

    Hello Flows!

    ; } ``` ## Configuration ### Change the port Edit `vite.config.ts`: ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} export default defineConfig({ server: { port: 3002, // Change from default 3001 }, }); ``` ### Disable auto-open Remove `fusionOpenPlugin` from `vite.config.ts` if you do not want the browser to open automatically: ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} export default defineConfig({ plugins: [ react(), mkcert(), // fusionOpenPlugin(), // Comment out ], }); ``` ## Troubleshooting ### App will not start **Check your Node.js version** Keep Node.js up to date. The generated `package.json` lists the versions the current template requires under `engines`. ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} node --version npm --version ``` If Node.js is outdated, update it using nvm (recommended): ```bash wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash . "$HOME/.nvm/nvm.sh" nvm install --lts nvm use --lts ``` See [Install Node.js](/cdf/flows/guides/getting-started#install-nodejs) in Get started for more detail. **`ERR_REQUIRE_ESM` error** If you see an error about `require()` of an ES module, your Node.js version is too old. Update Node.js and compare it with `engines` in `package.json`. **Dependencies not installed** ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm install ``` ### Certificate errors If `npm run dev` starts but the browser shows **Not Secure** or the app fails to load inside CDF, the dev server's HTTPS certificate isn't trusted by your browser. See [Trust the local HTTPS certificate](/cdf/flows/guides/local-https) for the one-time `mkcert` setup that fixes this on macOS, Linux, and Windows. ### Port already in use ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} lsof -i :3001 kill -9 ``` Or change the port in your Vite configuration. ## Further reading * [Get started with Flows](/cdf/flows/guides/getting-started) — Create an app and prerequisites. * [Deploy your Flows app](/cdf/flows/guides/deploying) — Interactive deploy and CI/CD. * [Auth API](/cdf/flows/reference/api/auth) — `connectToHostApp` and authenticated CDF access. * [Host App API](/cdf/flows/reference/api/host-app-api) — Navigation, state, sidebar visibility, fullscreen, and agent integration. * [Vite plugin API](/cdf/flows/reference/api/vite) — `fusionOpenPlugin` and development URLs. # Persisting app state in URLs Source: https://docs.cognite.com/cdf/flows/guides/shareable-app-state-with-urls Use syncInternalState to store your Flows custom app's view state in the URL, so users can share links that open the app exactly where they left off. This guide is for developers building Flows custom apps who want shareable deep links. Flows custom apps run inside an iframe and can't manage the browser URL directly. Call `syncInternalState` to store your app's view state in the `customAppInternalState` URL search param. This makes the URL shareable. Anyone who opens it sees the same view. Cognite Data Fusion (CDF) passes the stored value back as `initialState` the next time the app mounts. ## Prerequisites Before you start, ensure you have: * A **Flows** custom app created and running locally. See [Getting started with Flows custom apps](/cdf/flows/guides/getting-started). * **`@cognite/app-sdk`** installed. ## How it works When your app calls `syncInternalState`, CDF serializes your state object into the `customAppInternalState` URL search param. The state can capture anything that defines where the user was. That includes not just which page is open, but also component-level detail: which panels are collapsed, what values are prefilled in a form, which rows are expanded in a table, what filters are active. For example: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "activeTab": "trends", "selectedAssetId": "7518432", "sidebarCollapsed": true, "filters": { "status": "active", "minValue": 42 }, "prefilledForm": { "tag": "21PT1234", "unit": "bar" } } ``` CDF URL-encodes that and appends it to the browser URL: ```http wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} https://your-cluster.fusion.cognite.com/my-project/flows/apps/my-app ?customAppInternalState=%7B%22activeTab%22%3A%22trends%22%2C%22selectedAssetId%22%3A...%7D ``` The user can copy that URL and share it. When someone else (or the same user after navigating away) opens the link, CDF decodes the param and passes it back to your app as `initialState`. The app opens with the same tab, filters, and form values already in place. ## Implement state syncing Destructure `initialState` from `connectToHostApp`. If it exists, parse it and restore your app's state before the first render. ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { useEffect, useState } from "react"; import { connectToHostApp, type HostAppAPI } from "@cognite/app-sdk"; function App() { const [api, setApi] = useState(null); const [activeTab, setActiveTab] = useState("overview"); useEffect(() => { connectToHostApp() .then(({ api, initialState }) => { setApi(api); if (initialState) { try { setActiveTab(JSON.parse(initialState).activeTab); } catch { // ignore malformed state } } }) .catch(() => { // Not running inside CDF — continue with defaults }); }, []); // ... } ``` Always wrap `JSON.parse` in a `try`/`catch`. Saved state may be malformed if the schema changed between deploys. Open your app with a `customAppInternalState` value in the URL (or reload after syncing state in step 2). The restored tab, filters, and other fields match the saved state instead of your defaults. Call `syncInternalState` after any user action that changes meaningful state. Pass a JSON-serialized string. ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} function handleTabChange(tab: string) { setActiveTab(tab); void api?.syncInternalState(JSON.stringify({ activeTab: tab })); } ``` Pass the complete state object on every call. `syncInternalState` replaces the stored value; it does not merge. Change a persisted value (for example, switch tabs). The browser URL updates with a new `customAppInternalState` search param that reflects your change. ## Complete example Because `syncInternalState` **replaces** the stored value on every call, the safest approach is to keep all persisted fields in a single state object and update them together. That way there is one place to serialize from and one place to restore to. No field silently overwrites another. ```tsx wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { useEffect, useRef, useState } from "react"; import { connectToHostApp, type HostAppAPI } from "@cognite/app-sdk"; interface PersistedState { activeTab: string; sidebarCollapsed: boolean; filters: { status: string }; } const DEFAULT_STATE: PersistedState = { activeTab: "overview", sidebarCollapsed: false, filters: { status: "all" }, }; export function App() { const [api, setApi] = useState(null); const [state, setState] = useState(DEFAULT_STATE); const apiRef = useRef(null); useEffect(() => { connectToHostApp() .then(({ api, initialState }) => { apiRef.current = api; setApi(api); if (initialState) { try { setState({ ...DEFAULT_STATE, ...JSON.parse(initialState) }); } catch { // malformed — keep defaults } } }) .catch(() => { // standalone dev — continue without host }); }, []); function updateState(patch: Partial) { setState((prev) => { const next = { ...prev, ...patch }; void apiRef.current?.syncInternalState(JSON.stringify(next)); return next; }); } return ( <> updateState({ sidebarCollapsed: collapsed })} /> updateState({ activeTab: tab })} /> updateState({ filters })} /> ); } ``` `updateState` merges the patch into the current state and immediately syncs the full object. No field is ever lost. ## What to include in persisted state Include state that defines the user's current view and that they would want to return to: | Include | Exclude | | - | - | | Active tab or page | Loading flags (`isLoading`, `isFetching`) | | Selected item IDs | Server-fetched data (re-fetch on mount) | | Applied filters and sort order | Truly ephemeral UI state (open dropdown, tooltip) | | Search queries | Error messages | | Collapsed / expanded panel state | Sensitive values (tokens, passwords) | | Prefilled form values | Large blobs or binary data | | Zoom level or scroll position | N/A | Keep persisted state small. Store IDs and view parameters, not full data objects. Fetch data from the API on mount using the restored IDs. Browsers impose a maximum URL length (typically around 2,000 characters, though it varies by browser and server). Because the state is URL-encoded, large state objects can exceed this limit and cause the URL to be truncated or rejected. Avoid storing large state objects. If your serialized state grows beyond a few hundred characters, store only the minimal keys needed to reconstruct the view. ## Further reading * [Auth API](/cdf/flows/reference/api/auth) — `connectToHostApp` and authenticated CDF access. * [Host App API](/cdf/flows/reference/api/host-app-api) — `syncInternalState` and other interaction methods. * [Integrating AI agents with Flows](/cdf/flows/guides/ai_agent_integration) — Share app state with the built-in agent. * [Run your app locally](/cdf/flows/guides/running-locally) — Dev server, HTTPS, and CDF iframe. # Plan a dashboard app with spec-kit Source: https://docs.cognite.com/cdf/flows/guides/spec-kit-walkthrough Learn to use spec-kit by running specify, clarify, and plan on a real Production Accounting dashboard — from a short description to a reviewable implementation plan. You'll run the first three spec-kit commands — `/speckit.specify`, `/speckit.clarify`, and `/speckit.plan` — to plan a realistic Production Accounting dashboard. You will end with a `spec.md` that captures your intent, a Clarifications section that resolves every ambiguity, and a `plan.md` your agent can act on. The last two commands, `/speckit.tasks` and `/speckit.implement`, are not covered. See [Spec-driven development](/cdf/flows/concepts/spec-driven-development) for the full flow. The spec you write in this walkthrough drives toward a finished app like the one below — a KPI bar and a Balance Reconciliation Tree with quality badges, running as a Flows app in development mode: Production Accounting dashboard showing a KPI bar with Gross Production, Net Export, Imbalance, and Closure percent, above a Balance Reconciliation Tree of flow components with volume, source tag, and Verified, Estimated, or Manual quality badges. ## Prerequisites You need a **Flows** custom app with spec-kit enabled. If you don't have one yet: 1. Follow [Getting started with Flows custom apps](/cdf/flows/guides/getting-started) to create your app. 2. When prompted **Enable spec-driven development**, choose **Yes**, then choose **Yes** again when prompted to enable spec-kit. Once your app is open in Cursor or Claude Code, the spec-kit slash commands are available in `.cursor/commands/` and `.claude/commands/`. ## Step 1: Specify your feature Start with a short description of the app, then let spec-kit expand it into a structured spec. Give spec-kit a short description of what you want to build. You don't need to fill in every detail — the template will prompt you for the rest: ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} /speckit.specify Build an Oil Balance Detail dashboard called 'Production Accounting'. Landing page shows a KPI bar (Gross Production, Net Export, Imbalance, Closure %) and a Balance Reconciliation Tree — hierarchical table of flow components with values, source tags, and quality badges (Verified/Estimated/Manual). Read flow components and KPI inputs from CDF data models. React + Tailwind. ``` The command creates `specs/001-production-accounting/spec.md` pre-populated with the spec-kit template. Keep the one-liner focused on the surface and the key behavior. You will fill in edge cases and constraints during `/speckit.clarify`. Open `specs/001-production-accounting/spec.md`. The template has generated User Stories, acceptance scenarios, and placeholder Functional Requirements from your description. An example of the populated output (trimmed): ```markdown wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Feature Specification: Production Accounting Dashboard **Feature Branch**: `001-production-accounting` **Created**: 2026-06-08 **Status**: Draft **Input**: User description: "Build an Oil Balance Detail dashboard called 'Production Accounting'..." ## User Scenarios & Testing ### User Story 1 — View at-a-glance production KPIs (Priority: P1) A production accountant opens the dashboard to get an immediate snapshot of overall oil balance health. The KPI bar surfaces four headline metrics — Gross Production, Net Export, Imbalance, and Closure % — so the accountant can immediately identify if the period is closing within acceptable tolerance. **Acceptance Scenarios**: 1. **Given** the dashboard loads, **When** the landing page is displayed, **Then** four KPI cards appear in a horizontal bar: Gross Production (Mbbl), Net Export (Mbbl), Imbalance (Mbbl), and Closure (%). 2. **Given** Imbalance is outside the acceptable threshold, **Then** the Imbalance card renders in a warning or danger tone. ### User Story 2 — Explore the Balance Reconciliation Tree (Priority: P2) The accountant needs to trace where production volumes come from and where they go. The Balance Reconciliation Tree is a hierarchical table listing all flow components (Wellhead Production → Field Gross → Terminal Receipts → Pipeline Export → Losses). Each row shows the component name, a numeric value with unit, a source tag, and a quality badge (Verified / Estimated / Manual). ## Requirements ### Functional Requirements - **FR-001**: The dashboard MUST display a KPI bar with four metrics: Gross Production (Mbbl), Net Export (Mbbl), Imbalance (Mbbl), and Closure (%). Closure % MUST equal Net Export / Gross Production × 100. - **FR-004**: The Balance Reconciliation Tree MUST render as a hierarchical table with collapsible parent rows. - **FR-006**: Quality badges MUST support exactly three states: Verified (success tone), Estimated (warning tone), Manual (neutral tone). - **FR-009**: The dashboard MUST load flow components and balance periods from the `production_accounting` data model and read the values behind each source tag from `cdf_cdm.CogniteTimeSeries:v1`. ``` Fill in any sections the template left as placeholders, especially **Data Models & CDF Integration** and **Success Criteria**. Complete these sections before running `/speckit.plan`. The plan command depends on them to generate an accurate implementation plan. ## Step 2: Clarify ambiguities Resolve the open questions in the spec before you plan, so the implementation rests on explicit decisions. ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} /speckit.clarify ``` The command reads your spec and runs an internal coverage scan across categories: functional scope, data model, integration, and edge cases. It then surfaces up to five questions that would block planning if left unanswered. For each question, your agent presents a recommendation and an options table. The following example is from the Production Accounting spec: ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} **Question 3 of up to 5** The spec doesn't define how Imbalance is derived. In oil balance accounting this matters because the value could be computed from the tree (Gross Production minus Net Export minus Losses) or it could be a standalone value independent of the tree rows. **Recommended:** Option B — derived from tree totals. This ensures the KPI bar and the reconciliation tree are mathematically consistent (Imbalance = Gross Production − Net Export − Losses), which is the standard industry definition and keeps the dashboard internally coherent. | Option | Description | |--------|-------------| | A | Independent value — KPI bar values are all standalone, no formula | | B | Derived: Imbalance = Gross Production − Net Export − sum of Losses nodes | | C | Derived from tree totals but formula is an implementation detail | Reply with the option letter, "yes" to accept the recommendation, or a short answer (≤5 words). ``` Reply with the option letter or accept the recommendation. Your agent writes the answer directly back into the `spec.md` **Clarifications** section: ```markdown theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ## Clarifications ### Session 2026-06-08 - Q: What is the initial expand state of the Balance Reconciliation Tree on load? → A: Top-level rows expanded, all children collapsed (progressive disclosure). - Q: Is the Imbalance warning threshold user-configurable or hardcoded? → A: Hardcoded constant (±5 Mbbl), not user-adjustable in v1. - Q: Is the Imbalance KPI value derived from tree totals or independent of the tree? → A: Derived — Imbalance = Gross Production − Net Export − sum of Losses nodes; KPI bar and tree MUST be consistent. - Q: Is the dashboard a standalone page or embedded in a Cognite Data Fusion iframe? → A: Embedded in CDF iframe — use `@cognite/app-sdk` (`connectToHostApp`) for host integration and layout. - Q: Which UI state should be persisted in the CDF URL? → A: None — dashboard always opens in its default state on reload. ``` When all questions are resolved, your agent outputs a summary: ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ## Clarification complete 5 of 5 questions asked and answered. Spec: specs/001-production-accounting/spec.md Sections touched: Functional Requirements (FR-001, FR-002, FR-007, FR-011, FR-012), Key Entities (Imbalance derivation), Assumptions, Clarifications. All categories resolved. Suggested next command: /speckit.plan ``` At this point, `spec.md` is complete and ready for planning. Commit it before moving on so the decisions are in git history. ## Step 3: Generate the implementation plan Turn the clarified spec into a concrete, reviewable plan your agent can act on. ```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} /speckit.plan ``` The command reads the clarified spec and generates `specs/001-production-accounting/plan.md`. For the Production Accounting dashboard, the output included the following sections (trimmed): ```markdown wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # Implementation Plan: Production Accounting Dashboard **Branch**: `001-production-accounting` | **Date**: 2026-06-08 ## Summary Replace the current boilerplate App.tsx with a Production Accounting dashboard embedded in the Cognite Data Fusion iframe. The dashboard renders a KPI bar computed from the oil balance dataset, and a hierarchical Balance Reconciliation Tree with expand/collapse rows, quality badges, and a quality filter. Data is read from CDF data models through `connectToHostApp`. ## Technical Context Language/Version: TypeScript 5.x + React 18.3 Primary Dependencies: @cognite/aura/components, @cognite/app-sdk, @cognite/sdk, Tailwind CSS 4, Vitest 4 Storage: CDF data models — production_accounting (FlowComponent, BalancePeriod) and cdf_cdm (CogniteTimeSeries) Testing: Vitest + @testing-library/react Target Platform: Cognite Data Fusion iframe (desktop, 1280px+ viewport) ## Project Structure src/ ├── __mocks__/ │ └── balanceData.ts ├── types/ │ └── balance.ts ├── services/ │ ├── BalanceDataService.ts │ └── CdfBalanceDataService.ts ├── state/ │ └── DashboardStateProvider.tsx ├── hooks/ │ ├── useDashboardStorage.ts │ └── useProductionAccountingViewModel.ts ├── components/ │ ├── KpiBar/ │ │ ├── KpiBar.tsx │ │ ├── KpiCard.tsx │ │ └── KpiBar.test.tsx │ ├── ReconciliationTree/ │ │ ├── ReconciliationTree.tsx │ │ ├── TreeRow.tsx │ │ ├── QualityBadge.tsx │ │ ├── QualityFilter.tsx │ │ └── ReconciliationTree.test.tsx │ └── ProductionAccountingPage.tsx ├── App.tsx └── App.test.tsx ``` The plan also includes a **Constitution Check** — a table verifying that the proposed implementation satisfies each rule in `AGENTS.md`, such as using `@cognite/aura/components` before raw HTML, following the ViewModel pattern, and writing tests for every hook and component that contains logic. Review `plan.md` with your team before starting implementation. If anything looks off, update `spec.md` and re-run `/speckit.plan`. Commit both files together. You don't have to use `/speckit.plan` to generate this file. You can write `plan.md` yourself or produce it through your agent's in-session plan mode — as long as the steps are numbered and ordered clearly enough for `/speckit.tasks` to split into a task list. ## Troubleshooting The slash commands ship only when you enable spec-kit during `apps create`. Confirm that `.cursor/commands/` and `.claude/commands/` contain `speckit.*` files. If they're missing, you created the app without spec-kit — there is no retrofit command, so re-scaffold the app and choose **Yes** at the spec-kit prompt, or manually copy the spec-kit files into your app. See [Spec-driven development](/cdf/flows/concepts/spec-driven-development#spec-kit). The command asks only about ambiguities it finds, up to five. If your `spec.md` is already specific, it may ask fewer questions or none. Review the coverage report it prints. If a category you care about is marked resolved but you disagree, add detail to `spec.md` and run `/speckit.clarify` again. The plan is only as good as the clarified spec. Update `spec.md` — especially the Functional Requirements and the Data Models & CDF Integration section — then re-run `/speckit.plan`. Running the command again regenerates the plan from the current spec. ## What's next You've run specify, clarify, and plan on a real feature and committed a `spec.md` and `plan.md` to your repository. Your agent now has the context it needs to implement the Production Accounting dashboard. The next two spec-kit commands decompose the plan into executable tasks and run them in order: * `/speckit.tasks` — writes an ordered `tasks.md` where each task maps back to a functional requirement. * `/speckit.implement` — runs the tasks in sequence, stopping at each one to confirm before continuing. See [Spec-driven development](/cdf/flows/concepts/spec-driven-development) for how these commands work and how they fit into the full flow. # Building Flows custom applications Source: https://docs.cognite.com/cdf/flows/index Build production-ready web applications for industrial applications with AI-native development tools and modern web technologies. **Cognite Flows** enables you to build production-grade web applications that integrate seamlessly with CDF. **Flows** provides an accessible development platform with built-in templates, authentication, and AI-enhanced capabilities for creating scalable industrial applications. ## Key capabilities * **Modern web platform**: Easy-to-use templates with full customization options. * **Built-in CDF integration**: Authentication, data access, and SDK utilities. * **AI-native development**: Pre-built skills and agent patterns for AI coding agents — including Claude Code, Cursor, GitHub Copilot, and more. * **Developer-friendly tools**: Hot reload, automated testing, and simple deployment. * **Enterprise ready**: Version control and access control with scalable architecture and CI/CD support. Create an app, install dependencies, and run it in CDF. Become a certified Flows custom apps builder for customer environments. Run the four skill-driven steps to certify an app for production deployment. Request Dev status to publish builder-signed apps on a non-production project. Scaffolding, auth, Vite, tests, deploy, and data fetching at a glance. UI framework for CDF and Flows custom app UI. Patterns and pitfalls for keeping Flows apps well-structured as they grow. Plan features in checked-in specs your team and agent both work from. Agent-agnostic AI skills that ship with every Flows custom app. Dev server, HTTPS, CDF iframe, and local configuration. Use the App Hosting OAuth relay for an additional Microsoft Entra ID sign-in from within a Flows custom app. Interactive deploy and CI/CD. # Flows Auth API Source: https://docs.cognite.com/cdf/flows/reference/api/auth Complete reference for connectToHostApp and authenticated Cognite Data Fusion (CDF) access in Flows custom apps. Connect your Flows custom app to the Cognite Data Fusion (CDF) host window and get credentials for authenticated SDK access. Call `connectToHostApp` once at app startup. It returns a `HostAppAPI` instance with methods to get the current project, base URL, and access token. Use those credentials to construct a `CogniteClient`. For navigation, state, sidebar visibility, fullscreen, and agent integration, see [Host App API](/cdf/flows/reference/api/host-app-api). ## Prerequisites * A Flows custom app created with [`npx @cognite/cli@latest apps create`](/cdf/flows/guides/getting-started) * Basic understanding of React hooks and `useEffect` ## connectToHostApp Connects your app to the CDF host window and returns a `HostAppAPI` instance. ```tsx theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { connectToHostApp } from "@cognite/app-sdk"; const { api } = await connectToHostApp(); ``` ### Return value | Property | Type | Description | | - | - | - | | `api` | `HostAppAPI` | API for communicating with the CDF host | | `initialState` | `string \| undefined` | Restored state from a previous session (if `syncInternalState` was used) | ### Example: connect and get project ```tsx theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { useEffect, useState } from "react"; import { connectToHostApp } from "@cognite/app-sdk"; function App() { const [project, setProject] = useState(null); const [isLoading, setIsLoading] = useState(true); const [error, setError] = useState(); useEffect(() => { let cancelled = false; connectToHostApp() .then(async ({ api }) => { if (cancelled) return; const proj = await api.getProject(); if (!cancelled) setProject(proj); }) .catch((err: unknown) => { if (cancelled) return; setError(err instanceof Error ? err.message : String(err)); }) .finally(() => { if (!cancelled) setIsLoading(false); }); return () => { cancelled = true; }; }, []); if (isLoading) return
    Loading...
    ; if (error) return
    Error: {error}
    ; return
    Project: {project}
    ; } ``` ## HostAppAPI credential methods The `HostAppAPI` instance returned by `connectToHostApp` includes methods for the current project, base URL, and access token. For navigation, state, sidebar visibility, fullscreen, and agent integration, see [Host App API](/cdf/flows/reference/api/host-app-api). ### getProject Returns the current CDF project name. ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} const project = await api.getProject(); ``` ### getBaseUrl Returns the base URL of the CDF cluster (for example, `https://greenfield.cognitedata.com`). ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} const baseUrl = await api.getBaseUrl(); ``` ### getAccessToken Returns the current access token for authenticating CDF API calls. ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} const token = await api.getAccessToken(); ``` ### Example: construct a CogniteClient Use `getProject`, `getBaseUrl`, and `getAccessToken` together to build an authenticated `CogniteClient` from `@cognite/sdk`: ```tsx theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { useEffect, useState } from "react"; import { connectToHostApp } from "@cognite/app-sdk"; import { CogniteClient } from "@cognite/sdk"; function App() { const [sdk, setSdk] = useState(null); const [isLoading, setIsLoading] = useState(true); const [error, setError] = useState(); useEffect(() => { let cancelled = false; connectToHostApp() .then(async ({ api }) => { if (cancelled) return; const [project, baseUrl] = await Promise.all([ api.getProject(), api.getBaseUrl(), ]); const client = new CogniteClient({ project, baseUrl, getToken: () => api.getAccessToken(), }); if (!cancelled) setSdk(client); }) .catch((err: unknown) => { if (cancelled) return; setError(err instanceof Error ? err.message : String(err)); }) .finally(() => { if (!cancelled) setIsLoading(false); }); return () => { cancelled = true; }; }, []); if (isLoading) return
    Loading...
    ; if (error) return
    Error: {error}
    ; // pass sdk down or use context } ``` ## Further reading * [Getting started with Flows custom apps](/cdf/flows/guides/getting-started) — Create your first app * [Run your app locally](/cdf/flows/guides/running-locally) — Development with authentication * [Signing in with an additional Microsoft Entra ID account](/cdf/flows/guides/oauth-relay) — Additional Microsoft Entra ID sign-in via the App Hosting OAuth relay * [Flows features](/cdf/flows/concepts/features) — TanStack Query integration with `connectToHostApp` # Content Security Policy for Flows custom apps Source: https://docs.cognite.com/cdf/flows/reference/api/csp Understand the Content Security Policy (CSP) for Flows custom apps and how to allow trusted network sources in manifest.json. When you deploy a Flows custom app, the platform applies a strict [Content Security Policy (CSP)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) — a browser security feature that controls which external resources the app is allowed to load. CSP limits where scripts, styles, images, fonts, frames, workers, media, and network requests can come from. Most apps only need to change CSP when they call an external API, load map tiles, embed a frame, use a CDN, or add a dependency that creates workers or evaluates code at runtime. Add trusted sources in `permissions.network` in `manifest.json`. Run the app locally first. The [Flows Vite plugin](/cdf/flows/reference/api/vite) applies CSP during development and logs a manifest snippet when the browser blocks a request. ## How it works Use `manifest.json` to add trusted sources to supported fetch directives: ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["https://tiles.example.test"], "directives": ["img-src", "connect-src"] }, { "sources": ["https://api.partner.test"] } ] } } ``` If `directives` is omitted, the source is added to `connect-src`. Apps can't remove baseline restrictions, such as `object-src 'none'`, or widen document-level directives, such as `default-src` and `frame-ancestors`. Every Flows app starts with the production baseline. ## Debug CSP violations Run the app locally: ```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} npm run dev ``` When the browser blocks a request, the Vite plugin logs a `manifest.json` snippet to the browser console (DevTools → Console tab): ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} Flows Apps [linked-app-sdk v2] blocked connect-src -> https://tiles.example.test Update your manifest.json with the snippet below: { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["https://tiles.example.test"], "directives": ["connect-src"] } ] } } ``` Paste the rule into `manifest.json`. The dev server picks up the change without a restart. ## Production baseline Hosted production apps use this CSP baseline: ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} frame-ancestors https://*.fusion.cognite.com https://*.fusion.cogniteapp.com base-uri 'self' connect-src 'self' https://*.cognitedata.com wss://*.cognitedata.com default-src 'self' font-src 'self' data: https://fonts.gstatic.com form-action 'self' frame-src 'self' img-src 'self' data: blob: media-src 'self' object-src 'none' script-src 'self' style-src 'self' 'unsafe-inline' https://fonts.googleapis.com worker-src 'self' blob: ``` Important details: * CDF API and WebSocket endpoints are already allowed in `connect-src`. * Production `script-src` allows `'unsafe-inline'` to be added via `manifest.json`, but this is strongly discouraged. Use [hash sources](#inline-scripts-in-script-src) instead. * `eval` and `new Function(...)` are always blocked in production, regardless of `manifest.json`. * `style-src` includes `'unsafe-inline'` because many UI libraries inject style tags. * Aura font sources are already allowed. * `frame-ancestors` is controlled by App Hosting and can't be changed from `manifest.json`. ## Development differences Development uses the same baseline with a few additions for Vite and local framing: * `script-src` also includes `'unsafe-inline'` and `'unsafe-eval'` for Vite HMR and React Refresh. * `http://localhost`, `http://127.0.0.1`, `ws://localhost`, and `ws://127.0.0.1` sources are accepted during development. * Local and preview CDF origins can frame the app. These additions are removed in production. Dependencies that rely on development-only script relaxations will fail after deployment. ## Choose the right directive Use the browser CSP error to identify the blocked directive, then add the narrowest source that covers it. | Requirement | Directive | | - | - | | External HTTPS API | `connect-src` | | WebSocket | `connect-src` with a `wss://` source | | Map tiles, images, SVGs, or canvas textures | `img-src` | | Web fonts | `font-src` | | CDN stylesheet | `style-src` | | Embedded iframe | `frame-src` | | Audio or video | `media-src` | | Worker script from another origin | `worker-src` | | JavaScript file from another origin | `script-src` | Use `script-src` only for trusted script files. It can't approve `eval`, `new Function(...)`, or inline script tags in production. ## Manifest reference Each `permissions.network` entry has this shape: ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type NetworkRule = { sources: string[]; directives?: CspFetchDirective[]; }; ``` If `directives` is omitted, the source is added to `connect-src`. The valid values for `CspFetchDirective` are the eight directives in [Extensible directives](#extensible-directives). ### Extensible directives Only these fetch directives can be extended: ```text wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} connect-src, font-src, frame-src, img-src, media-src, script-src, style-src, worker-src ``` Rules that only target unsupported directives are dropped. Unsupported examples include `default-src`, `frame-ancestors`, `object-src`, `base-uri`, and `form-action`. ### Source formats Production accepts: * HTTPS sources, for example `https://api.example.test` or `https://api.example.test:8443/v1` * Secure WebSocket sources, for example `wss://stream.example.test` * Host wildcards, for example `https://*.example.test` Development also accepts local HTTP and WebSocket sources: * `http://localhost:3000` * `http://127.0.0.1:3000` * `ws://localhost:5173` * `ws://127.0.0.1:5173` For `script-src`, production also accepts: * `'wasm-unsafe-eval'` * `'strict-dynamic'` * Hash sources such as `'sha256-...'`, `'sha384-...'`, and `'sha512-...'` The policy rejects sources with whitespace, control characters, semicolons, stray quotes, `javascript:` URLs, `data:` URLs, and broad wildcards, such as `https://*`. Nonces are not supported — the Flows custom apps service does not generate per-request nonces. `manifest.json` can't add `'unsafe-eval'` to production `script-src`. Adding `'unsafe-inline'` is technically allowed but strongly discouraged — use a hash source to allowlist only the specific script you need, or move the code to an external file bundled by Vite. See [inline scripts](#inline-scripts-in-script-src). ## Examples ### External API ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["https://api.partner.test"] } ] } } ``` Because `directives` is omitted, the source is added to `connect-src`. ### Map tiles ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["https://tiles.example.test"], "directives": ["img-src", "connect-src"] } ] } } ``` Some map libraries fetch tile metadata with `fetch()` and load tile images through `` or canvas, so they need both directives. ### CDN stylesheet ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["https://cdn.example.test"], "directives": ["style-src"] } ] } } ``` ### WebAssembly ```json theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} { "manifestVersion": 1, "permissions": { "network": [ { "sources": ["'wasm-unsafe-eval'"], "directives": ["script-src"] } ] } } ``` Add this only when a library needs WebAssembly compilation at runtime. ## Common library issues Some libraries use runtime patterns that production CSP blocks. ### Libraries that use `eval` or `new Function` Templating, expression, charting, and schema libraries sometimes compile strings into functions at runtime. That requires `'unsafe-eval'` in `script-src`, which production apps don't allow. Use a build-time compiler, a precompiled distribution, or a different dependency. ### `react-pdf` and `pdfjs-dist` Configure the PDF worker as a same-origin asset bundled by Vite: ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { pdfjs } from "react-pdf"; pdfjs.GlobalWorkerOptions.workerSrc = new URL( "pdfjs-dist/build/pdf.worker.min.mjs", import.meta.url ).toString(); ``` The URL resolves to the app's own origin, which is covered by `'self'` in `worker-src`. ### `troika-three-text` and `reagraph` `troika-three-text`, used by `reagraph`, probes worker support with a blob worker and can call `importScripts()` with generated URLs. If that path is blocked, force troika onto the main thread before any `Text` instances or reagraph renders are created: ```ts theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} import { configureTextBuilder } from 'troika-three-text'; configureTextBuilder({ useWorker: false }); ``` ### Inline scripts in `script-src` Adding `'unsafe-inline'` to `script-src` allows any inline `