> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognite.com/llms.txt > Use this file to discover all available pages before exploring further. # YAML reference library > The YAML resource configuration files are core to the Cognite Toolkit, with each file configuring one of the resource types supported by the Cognite Toolkit and the Cognite API. > For the complete Cognite documentation index, see [/llms.txt](/llms.txt). For full page content, see [/llms-full.txt](/llms-full.txt). The YAML **resource configuration files** are core to the Cognite Toolkit. Each of the files configures one of the **resource types** that are supported by the Cognite Toolkit and the Cognite API. This article describes how to configure the different resource types. The Cognite Toolkit bundles logically connected resource configuration files in **modules**, and each module stores the configuration files in directories corresponding to the resource types, called **resource directories**. The available resource directories are: ```shell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} / 3dmodels/ agents/ (alpha) apps/ (alpha) auth/ cdf_applications/ classic/ data_modeling/ data_sets/ extraction_pipelines/ files/ functions/ hosted_extractors/ locations/ raw/ robotics/ rulesets/ (alpha) signals/ simulators/ streamlit/ timeseries/ transformations/ workflows/ ``` A resource directory can host one or more configuration types. For example, the `data_modeling/` directory hosts the configuration files for spaces, containers, views, data models, and nodes. While the `classic/` directory hosts the configuration files for labels, assets, and sequences. When you deploy, the Cognite Toolkit uses the Cognite API to implement the YAML configurations in the CDF project. ```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} flowchart LR A[(YAML file)]-->B>Cognite API] B-->C(CDF project) ``` In general, the format of the YAML files matches the [API specification](/api-reference/concepts/20230101/api-description) for the resource types. We recommend that you use the `externalId` of the resources as (part of) the name of the YAML file. This is to enable using the same resource configuration across multiple CDF projects, for example, a development, staging and production project. Use number prefixes (*1.\*) to control the order of deployment within each resource type. ## 3D Models **Resource directory: *3dmodels/*** *Requires Cognite Toolkit v0.3.0 or later* **API documentation:** [3D models](/api-reference/concepts/20230101/3d-models) 3D model configurations are stored in the module's *3dmodels/* directory. You can have one or more 3D models in a single YAML file. The filename must end with `3DModel`, for example, `my_3d_model.3DModel.yaml`. Example 3D model configuration: ```yaml title="my_3d_model.3DModel.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: my_3d_model dataSetExternalId: ds_3d_models metadata: origin: cognite-toolkit ``` The 3D model API uses an **internal ID** for the `dataSetId`, while the YAML configuration files reference the **external ID**; `dataSetExternalId`. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the Cognite API. ## Agents **Resource directory: *agents/*** *Requires Cognite Toolkit v0.5.85 or later and alpha flag `agents` enabled* **API documentation:** [Agents](/cdf/explore/agents) Agent configurations are stored in the module's *agents/* directory. You can have one or more agents in a single YAML file. It's recommended to have one agent per file. The filename must end with `Agent`, for example, `my_agent.Agent.yaml`. Example agent configuration: ```yaml title="my_agent.Agent.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_agent name: My Agent description: This is my agent instructions: This is instructions the agent. model: azure/gpt-4o-mini tools: - type: askDocument name: Document Search description: Search for information in documents. skills: - my_skill ``` To attach skills to an agent, list the skill `externalId` values in the `skills` field. Deploy the skill resources before the agent that references them. ### Available tools This section lists the available tools that you can use in the agent configuration. Note that this is a service in rapid development, and there may be more tools available than what is listed here. When you use a tool that the Cognite Toolkit doesn't recognize, you will get a warning when running the `cdf build` command. However, you can still deploy the tool to CDF as long as the CDF API supports the tool. **Ask document** ```yaml fragment="AskDocument" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ... - type: askDocument name: Document Search description: Search for information in documents. ``` **Summarize document** ```yaml fragment="SummarizeDocument" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - type: summarizeDocument name: Document Summarization description: Summarize the content of a document. ``` **Ask knowledge graph** ```yaml fragment="AskKnowledgeGraph" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - type: queryKnowledgeGraph name: Knowledge Graph Query description: Query the knowledge graph for information. configuration: dataModels: - space: cdf_cdm externalId: CogniteCore version: v1 viewExternalIds: - CogniteAsset - CogniteEquipment - CogniteTimeSeries - CogniteFile - CogniteActivity instanceSpaces: type: manual spaces: - my_space - my_other_space # Or you can include all instance spaces with # instanceSpaces: # type: all ``` **Query time series data points** ```yaml fragment="queryTimeSeriesDatapoints" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - type: queryTimeSeriesDatapoints name: Time Series Query description: Query time series data. ``` ## Agent skills **Resource directory: *agents/*** *Requires the `agent-skills` alpha flag enabled* **API documentation:** [Skills (beta)](https://api-docs.cognite.com/20230101-beta/tag/Skills/) Alpha flags enable experimental Cognite Toolkit behavior that may change or be removed without notice. Agent skills extend Atlas AI agents with reusable instructions in [Agent Skills](https://agentskills.io/specification) `SKILL.md` format. Skill configurations are stored in the module's *agents/* directory alongside agent definitions. The filename must end with `Skill`, for example, `my_skill.Skill.yaml`. Each skill uses a minimal YAML file for identity and a Markdown sidecar for the skill body. The sidecar must include YAML frontmatter with `name` and `description`, followed by a Markdown body. The `name` and `description` in the frontmatter must match the values sent to the API. ### Skill configuration ```yaml title="my_skill.Skill.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_skill ``` Store the skill instructions in a sidecar Markdown file next to the YAML file. The Cognite Toolkit looks for content in this order: 1. A sibling file with the same basename and a `.md` extension (for example, `my_skill.Skill.md` next to `my_skill.Skill.yaml`). 2. A file named `{externalId}.Skill.md` in the same directory (for example, `my_skill.Skill.md`). ```markdown title="my_skill.Skill.md" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} --- name: my-skill description: Summarize equipment downtime reports and suggest follow-up actions. --- # Equipment downtime summary When the user asks about equipment downtime: 1. Query the knowledge graph for recent activity records. 2. Summarize outages by duration and root cause. 3. Suggest follow-up maintenance actions. ``` Alternatively, you can include the full `SKILL.md` content inline in the YAML file: ```yaml title="my_skill.Skill.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_skill content: | --- name: my-skill description: Summarize equipment downtime reports and suggest follow-up actions. --- # Equipment downtime summary When the user asks about equipment downtime: 1. Query the knowledge graph for recent activity records. 2. Summarize outages by duration and root cause. 3. Suggest follow-up maintenance actions. ``` The skill `name` in the Markdown frontmatter follows the [Agent Skills](https://agentskills.io/specification) naming rules: lowercase letters, digits, and hyphens only. The `externalId` in the YAML file is the CDF identifier and can use a different format, such as underscores. ## Assets **Resource directory: *classic/*** *Requires Cognite Toolkit v0.3.0 or later* **API documentation:** [Assets](/api-reference/concepts/20230101/assets) Asset configurations are stored in the module's *classic/* directory. You can have one or more asset in a single YAML file. The filename must end with `Asset`, for example, `my_asset.Asset.yaml`. Example Asset configuration: ```yaml title="my_root.Asset.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_root_asset name: SAP hierarchy description: The root asset in the SAP hierarchy dataSetExternalId: ds_sap_assets source: SAP metadata: origin: cognite-toolkit ``` The asset API uses an **internal ID** for the data set, while the YAML configuration files reference the **external ID**; `dataSetExternalId`. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the Cognite API. The Cognite Toolkit supports all fields in the Asset API. See below for a complete example: ```yaml title="my_asset.Asset.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_asset name: My asset parentExternalId: my_root_asset description: This is my asset dataSetExternalId: ds_assets source: SAP metadata: origin: cognite-toolkit my_custom_field: my_custom_value labels: externalId: my_label ``` The CogniteToolkit doesn't support the field `parentId` in the asset configuration. Instead, use `parentExternalId`. The `parentId` field is an **internal ID** that is not consistent across environments. ### Table formats In addition to `yaml`, the Cognite Toolkit supports `csv` and `parquet` formats for asset configurations. As with the `yaml` format, the filename must end with `Asset`, for example, `my_asset.Asset.csv`. ```csv title="my_root.Asset.csv" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId,name,description,dataSetExternalId,source,metadata.origin my_root_asset,SAP hierarchy,The root asset in the SAP hierarchy,ds_sap_assets,SAP,cognite-toolkit ``` The column names must match the field names in the `yaml` configuration. The exception is the `metadata` field, which is a dictionary in the `yaml` configuration, but a string in the `csv` configuration. This is solved by using the notation `metadata.origin` column in the `csv` configuration. ## Groups **Resource directory: *auth/*** **API documentation:** [Groups](/api-reference/concepts/20230101/groups) The group configuration files are stored in the module's *auth/* directory. You can have one or more groups in a single YAML file. The filename must end with `Group`, for example, `my_group.Group.yaml`. The `name` field is used as a unique identifier for the group. If you change the name of the group manually in CDF, it will be treated as a different group and will be ignored by the Cognite Toolkit. **Example group configuration:** ```yaml title="my_group.Group.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: 'my_group' sourceId: '{{mygroup_source_id}}' metadata: origin: 'cognite-toolkit' capabilities: - projectsAcl: actions: - LIST - READ scope: all: {} ``` We recommend using the `metadata:origin` property to indicate that the group is created by the Cognite Toolkit You can specify each ACL capability in CDF as in the `projectsAcl` example above. Scoping to dataset, space, RAW table, current user, or pipeline is also supported (see [ACL scoping](#ACL-scoping)). ### Groups and group deletion If you delete groups with the `cdf clean` or `cdf deploy --drop` command, the Cognite Toolkit skips the groups that the running user or service account is a member of. This prevents the cleaning operation from removing access rights from the running user and potentially locking the user out from further operation. ### ACL scoping #### Dataset scope Use to restrict access to data in a specific **data set**. ```yaml title="" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - threedAcl: actions: - READ scope: datasetScope: { ids: ['my_dataset'] } ``` The groups API uses an **internal ID** for the data set, while the YAML configuration files reference the **external ID**. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the Cognite API. #### Space scope Use to restrict access to data in a **data model space**. ```yaml title="" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - dataModelInstancesAcl: actions: - READ scope: spaceIdScope: { spaceIds: ['my_space'] } ``` #### Table scope Use to restrict access to a **database**. ```yaml title="" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - rawAcl: actions: - READ - WRITE scope: tableScope: dbsToTables: my_database: [] ``` Use to restrict access to a **table** in a database. ```yaml title="" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - rawAcl: actions: - READ - WRITE scope: tableScope: dbsToTables: my_database: ["my_table", "my_other_table"] ``` The CDF API also supports `my_database: {"tables": []}` and `my_database: {"tables": ["my_table"]}` to restrict access to all tables in a database or a specific table in a database. The Cognite Toolkit will warn you if you use these notations, because the `cdf dump group` and `cdf modules pull` commands work better with consistent YAML files. The toolkit recommends using only the list notation. #### Current user-scope Use to restrict actions to the **groups the user is a member of**. ```yaml title="" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - groupsAcl: actions: - LIST - READ scope: currentuserscope: {} ``` ## Security categories *Requires Cognite Toolkit v0.2.0 or later* **Resource directory: *auth/*** **API documentation:** [Security categories](/api-reference/concepts/20230101/security-categories) The security categories are stored in the module's *auth/* directory. You can have one or more security categories in a single YAML file. We recommend that you start security category names with `sc_` and use `_` to separate words. The file name is not significant, but we recommend that you name it after the security categories it creates. ```yaml title="project_categories.SecurityCategory.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - name: sc_my_security_category - name: sc_my_other_security_category ``` ## Data models **Resource directory: *data\_modeling/***. Note that earlier versions of the Cognite Toolkit used the *data\_models/* directory. **API documentation:** [Data modeling](/api-reference/concepts/20230101/data-modeling) The data model configurations are stored in the module's **data\_modeling** directory. A data model consists of a set of data modeling entities: one or more [spaces](#spaces), [containers](#containers), [views](#views), and [data models](#data-models). Each entity has its own file with a suffix to indicate the entity type: *my.space.yaml*, *my.container.yaml*, *my.view\.yaml*, *my.datamodel.yaml*. You can also use the Cognite Toolkit to create [nodes](#nodes) to keep configuration for applications (for instance, InField) and to create node types that are part of the data model. Define nodes in files with the *.node.yaml* suffix. The Cognite Toolkit applies configurations in the order of dependencies between the entity types: first **spaces**, next **containers**, then **views**, and finally **data models**. If there are dependencies between the entities of the same type, you can use prefix numbers in the filename to have the Cognite Toolkit apply the files in the correct order. The Cognite Toolkit supports using subdirectories to organize the files, for example: ```shell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} data_modeling/ containers/ views/ nodes/ my_data_model.datamodel.yaml data_model_space.space.yaml ``` ### Spaces **API documentation:** [Spaces](/api-reference/concepts/20230101/spaces) Spaces are the top-level entities and is the home of containers, views, and data models. You can create a space with a *.space.yaml* file in the *data\_modeling/* directory. ```yaml title="sp_cognite_app_data.space.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} space: sp_cognite_app_data name: cognite:app:data description: Space for InField app data ``` CDF doesn't allow a space to be deleted unless it's empty. If a space contains, for example, nodes that aren't governed by the Cognite Toolkit, the Cognite Toolkit will not delete the space. ### Containers **API documentation:** [Containers](/api-reference/concepts/20230101/containers) Containers are the home of properties and data. You can create a container with a *.container.yaml* file in the *data\_modeling/* directory. You can also create indexes and constraints according to the API specification. ```yaml title="MyActivity.container.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: MyActivity usedFor: node space: sp_activity_data description: 'A container for activity data.' name: The display name of the container properties: id: type: type: text list: false collation: ucs_basic nullable: true immutable: false autoIncrement: false title: type: type: text list: false collation: ucs_basic nullable: true defaultValue: No title description: The title of the activity name: Display name of the title property description: type: type: text list: false collation: ucs_basic nullable: true asset: type: type: direct list: false nullable: true description: The asset the activity is related to indexes: idLookup: properties: - id indexType: btree cursorable: true constraints: idUnique: constraintType: uniqueness properties: - id ``` The example container definition creates a container with the properties `id`, `title`, `description`, and `asset`. Note that `sp_activity_data` requires its own *activity\_data.space.yaml* file in the *data\_modeling/* directory. ### Views **API documentation:** [Views](/api-reference/concepts/20230101/views) Use views to ingest, query, and structure the data into meaningful entities in your data model. You can create a view with a *.view\.yaml* file in the *data\_modeling/* directory. ```yaml title="MyActivity.view.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: MyActivity name: MyActivity description: 'An activity represents a set of maintenance tasks with multiple operations for individual assets. The activity is considered incomplete until all its operations are finished.' version: '3' space: sp_activity_model implements: - type: view externalId: CogniteActivity space: cdf_cdm version: v1 properties: id: description: 'Unique identifier from the source, for instance, an object ID in SAP.' container: type: container space: sp_activity_data externalId: MyActivity containerPropertyIdentifier: id title: description: 'A title or brief description of the maintenance activity or work order.' container: type: container space: sp_activity_data externalId: MyActivity containerPropertyIdentifier: title description: description: 'A detailed description of the maintenance activity or work order.' container: type: container space: sp_activity_data externalId: MyActivity containerPropertyIdentifier: description asset: description: 'The asset the activity is related to.' container: type: container space: sp_activity_data externalId: MyActivity containerPropertyIdentifier: asset source: type: view externalId: MyAsset space: sp_activity_model version: '2' ``` This example view configuration creates a view with the properties `id`, `title`, `description`, and `asset`. The `asset` property references the `MyAsset` view in the `sp_activity_model` space. The view references the properties from the container `MyActivity` in the `sp_activity_data` space. The view exists in a space called `sp_activity_model`, while the container exists in the `sp_activity_data` space. ### Data models **API documentation:** [Data models](/api-reference/concepts/20230101/data-models) Use data models to structure the data into knowledge graphs with relationships between views using edges. From an implementation perspective, a data model is a collection of views. You can create a data model with a *.datamodel.yaml* file in the *data\_modeling/* directory. ```yaml title="ActivityDataModel.datamodel.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: ActivityDataModel name: My activity data model version: '1' space: sp_activity_model description: 'A data model for structuring and querying activity data.' views: - type: view externalId: MyActivity space: sp_activity_model version: '3' - type: view externalId: MyTask space: sp_activity_model version: '2' ``` The example data model configuration creates a data model with two views: `MyActivity` and `MyTasks`. The data model exists in a space called `sp_activity_model` together with the views. ### Nodes **API documentation:** [Instances](/api-reference/concepts/20230101/instances) Use nodes to populate a data model. You can create nodes with a *.node.yaml* file in the *data\_modeling/* directory. ```yaml title="myapp_config.node.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: sp_config externalId: myapp_config type: space: sp_config_schema externalId: ConfigNode sources: - source: space: sp_config externalId: MY_APP_Config version: '1' type: view properties: rootLocationConfigurations: - assetExternalId: 'my_root_asset_external_id' adminGroup: - gp_template_admins dataSpaceId: sp_activity_data modelSpaceId: sp_activity_model activityDataModelId: MyActivity activityDataModelVersion: '1' ``` This example node configuration creates a node instance with data that configures a node of the type `MY_APP_Config` with version '1' in the `sp_config` space. The instance has data that is read by MY\_APP and used to configure the application. The node instance is created in the `sp_config` space with `myapp_config` as the `externalId`. The example also configures a root location for the application and specifies how to find the application's data: in the `sp_activity_data` space with version `1` of the `MyActivity`view. Another example is **node types**. They are part of a data model schema (the description of how data is structured), and creates a type of node that can be created in the data model. This is an example of a YAML file with multiple node types defined. ```yaml title="myapp_config.node.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: sp_my_model externalId: pump - space: sp_my_model externalId: valve - space: sp_config_schema externalId: ConfigNode ``` ### Edges *Requires Cognite Toolkit v0.4.0 or later* **API documentation:** [Instances](/api-reference/concepts/20230101/instances) Use edges to define connections between nodes in a data model. You can create edges with a *.edge.yaml* file in the *data\_modeling/* directory. You can have one or more edges in a single YAML file. The filename must end with `Edge`, for example, `my_edge.Edge.yaml`. ```yaml title="my_edge.Edge.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} space: sp_instance externalId: 'MyEdge' startNode: space: sp_instance externalId: 'startNode' endNode: space: sp_instance externalId: 'endNode' type: space: sp_schema externalId: 'AnEdgeType' sources: - source: space: sp_schema externalId: 'MyView' version: v1 type: 'view' properties: myProperty: 'myValue' ``` ## Data sets **Resource directory: *data\_sets/*** **API documentation:** [Data sets](/api-reference/concepts/20230101/data-sets) You can not delete data sets in CDF, but you can use the Cognite Toolkit to create new data sets or update existing ones. You can create multiple data sets in the same YAML file. The filename must end with `DataSet`, for example, `my_data_set.DataSet.yaml`. The data sets API uses an **internal ID** for the data set, while the YAML configuration files reference the **external ID**; `dataSetExternalId`. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the Cognite API. For an example, see [files](#files-dir-files). ```yaml title="data_sets.DataSet.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: ds_asset_hamburg name: asset:hamburg description: This dataset contains asset data for the Hamburg location. writeProtected: false metadata: origin: cognite-toolkit - externalId: ds_files_hamburg name: files:hamburg description: This dataset contains files for the Hamburg location. writeProtected: false metadata: origin: cognite-toolkit ``` This example configuration creates two data sets using the [naming conventions](/cdf/deploy/reference/cdf_resource_naming_conventions#data-sets) for data sets. ## Events *Requires Cognite Toolkit v0.4.0 or later* **Resource directory: *classic/*** **API documentation:** [Events](/api-reference/concepts/20230101/events) Events can be found in the module's `classic/` directory. You can define one or more events in a single YAML file. The filename must end with `Event`, for example, `my_event.Event.yaml`. The Cognite Toolkit ensures that dependent resources are created before the events. For example, if you reference an asset or a data set in an event, the Cognite Toolkit creates the asset or data set before creating the event. Example event definition: ```yaml title="my_event.Event.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: MyEvent dataSetExternalId: ds_complete_org startTime: 1732959346052 endTime: 1732959346052 type: 'success' subtype: 'info' description: 'My event description' metadata: key: value assetExternalIds: - MyAsset - MyAsset2 source: 'my_source' ``` The data set is referenced by the `dataSetExternalId` and `assetExternalIds`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set and assets before sending the request to CDF. ## InfieldV1 *Requires Cognite Toolkit v0.5.10 or later and alpha flag `infield` enabled* **Resource directory: *cdf\_applications/*** InfieldV1 can be found in the modules `cdf_applications/` directory. You can define one or more InfieldV1 configurations in a single YAML file. The filename must end with `InfieldV1`, for example, `my_infield_app.InfieldV1.yaml`. The Cognite Toolkit ensures that dependent resources are created before the InfieldV1 applications. For example, if you reference a space, data sets, assets, or groups in the InfieldV1 configuration, the Cognite Toolkit creates it before creating the InfieldV1 configuration. Example InfieldV1 configuration: ```yaml title="my_infield_app.InfieldV1.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: default_infield_config_minimal featureConfiguration: rootLocationConfigurations: - assetExternalId: WMT:VAL appDataInstanceSpace: sp_infield_oid_app_data sourceDataInstanceSpace: sp_asset_oid_source templateAdmins: - gp_infield_oid_template_admins checklistAdmins: - gp_infield_oid_checklist_admins dataSetExternalId: ds_complete_org customerDataSpaceId: APM_SourceData customerDataSpaceVersion: '1' name: Infield APM App Config ``` The data set is referenced by the `dataSetExternalId`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. ## Infield location configuration *Requires Cognite Toolkit v0.7.105 or later and alpha flag `infield` enabled* **Resource directory: *cdf\_applications/*** You can find Infield location configurations in the modules `cdf_applications/` directory. You can define one or more Infield location configurations in a single YAML file. The filename must end with `InfieldLocationConfig`, for example, `my_location_config.InfieldLocationConfig.yaml`. Example Infield location configuration: ```yaml title="my_location_config.InfieldLocationConfig.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} space: sp_instance externalId: my_location_config_001 rootLocationExternalId: location_filter_main_facility featureToggles: threeD: true trends: true documents: true workorders: true notifications: true media: true templateChecklistFlow: true workorderChecklistFlow: false appInstanceSpace: my_app_instance_space accessManagement: templateAdmins: - cdf_group_template_admins - cdf_group_superusers checklistAdmins: - cdf_group_checklist_admins - cdf_group_superusers dataFilters: general: spaces: - my_space_001 - my_space_002 dataExplorationConfig: externalId: data_exploration_config_shared observations: enabled: true defaultView: list activities: enabled: true showCompleted: true documents: enabled: true supportedFormats: - pdf - docx notifications: enabled: true pushEnabled: false assets: enabled: true showHierarchy: true ``` Toolkit converts the configuration into a node that it uploads into the `cdf_infield, InFieldLocationConfig, v1` and the `cdf_infield, DataExplorationConfig, v1` views. ## Labels *Requires Cognite Toolkit v0.3.0 or later* **Resource directory: *classic/*** (*labels/* in `v0.2.0`) **API documentation:** [Labels](/api-reference/concepts/20230101/labels) Labels can be found in the module's `classic/` directory. You can define one or more labels in a single YAML file. The filename must end with `Label`, for example, `my_equipment.Label.yaml`. The Cognite Toolkit creates labels before files and other resources that reference them. Example label definition: ```yaml title="my_equipment.Label.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: label_pump name: Pump description: A pump is an equipment that moves fluids. dataSetExternalId: ds_labels_{{example_variable}} ``` The Cognite API doesn't support updating labels. When you update a label with the Cognite Toolkit, it deletes the previous label and creates a new one. ## Extraction pipelines **Resource directory: *extraction\_pipelines/*** **API documentation:** [Extraction pipelines](/api-reference/concepts/20230101/extraction-pipelines) **API documentation Configuration:** [Extraction pipeline config](/api-reference/concepts/20230101/extraction-pipelines-config) **Documentation:** [Extraction pipeline documentation](/cdf/integration/guides/interfaces/add_integrations) Extractor pipelines and configurations are stored in the module's *extraction\_pipelines/* directory. You can define one or more extraction pipelines in a single YAML file. It is, however, most common to have one pipeline per file. In addition, one or more the extraction pipeline configurations must be stored in a separate file. Extraction pipeline configurations are detected by the `.config` suffix, while extraction pipeline filenames end with `ExtractionPipeline`. ```yaml title="ep_src_asset_hamburg_sap.ExtractionPipeline.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'ep_src_asset_hamburg_sap' name: 'src:asset:hamburg:sap' dataSetExternalId: 'ds_asset_{{location_name}}' description: 'Asset source extraction pipeline with configuration for a DB extractor reading data from Hamburg SAP' rawTables: - dbName: 'asset_hamburg_sap' tableName: 'assets' source: 'sap' documentation: "The DB Extractor is a general database extractor that connects to a database, runs one or several queries, and sends the result to CDF RAW.\n\nThe extractor connects to a database over ODBC, which means that you need an ODBC driver for your database. If you are running the Docker version of the extractor, ODBC drivers for MySQL, MS SQL, PostgreSql and Oracle DB are preinstalled in the image. See the example config for details on connection strings for these. If you are running the Windows exe version of the extractor, you must provide an ODBC driver yourself. These are typically provided by the database vendor.\n\nFurther documentation is available [here](/cdf/deploy/cdf_toolkit/references/docs/documentation)\n\nFor information on development, consider the following guides:\n\n * [Development guide](/cdf/deploy/cdf_toolkit/references/guides/development)\n * [Release guide](/cdf/deploy/cdf_toolkit/references/guides/release)" schedule: Continuous contacts: - name: 'John Doe' email: john.doe@company.com role: 'Extractor developer' sendNotification: true metadata: origin: 'cognite-toolkit' my_custom_field: 'my_custom_value' notificationConfig: allowedNotSeenRangeInMinutes: 120 createdBy: 'John Doe' ``` This example configuration creates an extraction pipeline with the external ID `ep_src_asset_hamburg_sap` and the name `src:asset:hamburg:sap`. The configuration allows an extractor installed inside a closed network to connect to CDF and download the extractor's configuration file. The Cognite Toolkit expects the configuration file be in the same directory and have the same name as the extraction pipeline configuration file, but with the suffix `.config.yaml`. The configuration file is not strictly required, but the Cognite Toolkit warns if the file is missing during the deployment process. The extraction pipeline can be connected to a data set and to the RAW tables that the extractor will write to. This is an example configuration file for the extraction pipeline above: ```yaml title="ep_src_asset_hamburg_sap.config.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'ep_src_asset_hamburg_sap' description: 'DB extractor config reading data from Hamburg SAP' config: logger: console: level: INFO file: level: INFO path: 'file.log' # List of databases databases: - type: odbc name: postgres connection-string: 'DSN={MyPostgresDsn}' # List of queries queries: - name: test-postgres database: postgres query: > SELECT ``` The Cognite Toolkit expects the `config` property to be valid YAML and will not validate the content of the config property beyond the syntax validation. The extractor that is configured to download the configuration file validates the content of the `config` property. ## Files **Resource directory *files/*** **API documentation:** [Files](/api-reference/concepts/20230101/file-content) `CogniteFile` Requires Cognite Toolkit v0.3.0 or later Use the Cognite Toolkit only to upload **example data**, and not as a general solution to ingest files into CDF. Files can be found in the module's `files/` directory. You can define the metadata for one or more files using either a single YAML file or multiple YAML files. Mark the files as `CogniteFile` or `FileMetadata` by using the suffix `.CogniteFile.yaml` or `.FileMetadata.yaml`. Files with a suffix other than `.yaml`, such as `.txt`, or YAML files that do not end with `FileMetadata` or `CogniteFile`, are expected to be content of a configuration file. To upload a file with the metadata, the `name` in the `YAML` file must match the filename of the file that should be uploaded. You can also use the [template for uploading multiple files](#uploading-multiple-files) to upload multiple files without specifying the metadata for each file. Below is an example of a classic file metadata configuration for multiple files, `my_file.pdf` and `my_other_file.pdf`: ```yaml title="my_files.FileMetadata.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: 'sharepointABC_my_file.pdf' name: 'my_file.pdf' source: 'sharepointABC' dataSetExternalId: 'ds_files_hamburg' directory: 'files' mimeType: 'application/pdf' metadata: origin: 'cognite-toolkit' assetExternalIds: - 'my_root_asset' sourceCreatedTime: 1640995200000 sourceModifiedTime: 1640995200000 securityCategories: - 'sc_my_security_category' labels: - externalId: 'my_label' - externalId: 'sharepointABC_my_other_file.pdf' name: 'my_other_file.pdf' source: 'sharepointABC' dataSetExternalId: 'ds_files_hamburg' directory: 'files' mimeType: 'application/pdf' metadata: origin: 'cognite-toolkit' ``` Classic metadata configuration for a single file, `my_file.pdf`: ```yaml title="my_file.FileMetadata.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'sharepointABC_my_file.pdf' name: 'my_file.pdf' source: 'sharepointABC' dataSetExternalId: 'ds_files_hamburg' directory: 'files' mimeType: 'application/pdf' metadata: origin: 'cdf-project-templates' ``` The data set is referenced by the `dataSetExternalId`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. Below is an example of a `CogniteFile` metadata configuration for a single file, `my_file.pdf`: ```yaml title="my_file.CogniteFile.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'sharepointABC_my_file.pdf' space: 'sp_files_hamburg' name: 'my_file.pdf' description: 'This is a file uploaded from SharePoint.' tags: - 'file' - 'sharepoint' sourceId: 'sharepointABC' sourceContext: 'sharepointABC' source: space: 'sp_files_hamburg' externalId: 'sharepointABCSource' sourceCreatedTime: '2022-01-01T00:00:00Z' sourceUpdatedTime: '2022-01-01T00:00:00Z' assets: - space: 'sp_assets' externalId: 'my_root_asset' mimeType: 'application/pdf' directory: 'files' category: - space: 'sp_categories' externalId: 'sc_my_category' ``` Below is an example of a `CogniteFile` metadata configuration for multiple files, `my_file.pdf` and `my_other_file.pdf`: ```yaml title="my_files.CogniteFile.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: 'sp_files_hamburg' externalId: 'sharepointABC_my_file.pdf' name: 'my_file.pdf' description: 'This is a file uploaded from SharePoint.' - space: 'sp_files_hamburg' externalId: 'sharepointABC_my_other_file.pdf' name: 'my_other_file.pdf' description: 'This is another file uploaded from SharePoint.' ``` ### Uploading multiple files To upload multiple files without specifying the metadata configuration for each file individually, use this template format for the `FileMetadata` configuration file: ```yaml title="files.FileMetadata.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: sharepointABC_$FILENAME dataSetExternalId: ds_files_hamburg name: $FILENAME source: sharepointABC ``` or for the `CogniteFile` configuration file: ```yaml title="files.CogniteFile.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - space: 'sp_files_hamburg' externalId: sharepointABC_$FILENAME name: $FILENAME description: 'This is a file uploaded from SharePoint.' ``` This template is recognized by the Cognite Toolkit by * It is a YAML file given in `list/array` format. * There is a single entry in the list. * The `externalId` contains the `$FILENAME` variable. All files will be uploaded with the same properties except for the `externalId` and `name` properties. The `$FILENAME` variable will be replaced with the filename of the file being uploaded. ## Functions **Resource directory *functions/*** **API documentation:** [Functions](/api-reference/concepts/20230101/functions) The function configuration files are stored in the module's *functions/* directory. You can define one or more functions in a single or multiple YAML file(s). The Cognite Toolkit creates the functions in the order they are defined in the file. Functions files must end with `Function`, for example, `my_function.Function.yaml`. The functions YAML files must be located in the *functions/* directory and not in subdirectories. This allows you to store YAML files that are not configuration files in subdirectories as part of the function's code. Place the function code and files to deploy to CDF as a function in a subdirectory with the same name as the `externalId` of the function. **Example function configuration:** Folder structure, including a function schedule: ```shell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} functions/ my_function.yaml schedules.yaml fn_example_repeater/ handler.py requirements.txt ``` Configuration file: ```yaml title="my_functions.Function.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # The directory with the function code must have the same name # and externalId as the function itself as defined below. - name: example:repeater externalId: fn_example_repeater owner: Anonymous description: Returns the input data, secrets, and function info. metadata: version: '{{version}}' secrets: mysecret: {{example_secret}} envVars: # The two environment variables below are set by the Toolkit ENV_TYPE: ${CDF_BUILD_TYPE} CDF_ENV: ${CDF_ENVIRON} runtime: 'py311' functionPath: ./handler.py # Data set id for the zip file with the code that is uploaded. dataSetExternalId: ds_files_{{default_location}} ``` ```python title="fn_example_repeater/handler.py" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} from cognite.client import CogniteClient def handle(data: dict, client: CogniteClient, secrets: dict, function_call_info: dict) -> dict: print("Function 'fn_example_repeater' called with data:", data) return { "secrets": mask_secrets(secrets), "functionInfo": function_call_info, } def mask_secrets(secrets: dict) -> dict: return {k: "***" for k in secrets} ``` ```text title="fn_example_repeater/requirements.txt" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite-sdk ``` The `functionPath` is the path to the *handler.py* in the function code directory. In this case, *handler.py* is expected to be in the *fn\_example\_repeater/* directory. Note that `dataSetExternalId` is not part of the function API. The Cognite Toolkit uploads the function code to a `FileMetadata` and references the uploaded file when creating a Cognite Function. If the `dataSetExternalId` is set, the Cognite Toolkit uses it on the `FileMetadata` upload. If not set, the code file will be uploaded without a data set. Alternatively, you can define `space` to store the function code as a `CogniteFile` instead of a `FileMetadata`: ```yaml title="my_dm_function_code.Function.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} # The directory with the function code must have the same name # and externalId as the function itself as defined below. name: my_function externalId: my_function # This will enable storing the function code as a CogniteFile instead of FileMetadata. The Cognite Toolkit will # create a node in the specified by the space and the same externalId as the function. space: sp_function_code ``` ### Dependency management The `requirements.txt` file in the function code directory specifies the Python packages your function depends on. By default, packages are installed from [PyPI](https://pypi.org/simple), but you can also include private packages by configuring a custom package index or including wheel files directly in the function code directory. #### Using indexUrl and extraIndexUrls When deploying a Cognite Function, CDF resolves dependencies by first checking the `indexUrl`. If a package isn't found there, CDF checks the `extraIndexUrls` in order. By default, `indexUrl` points to `https://pypi.org/simple`. If your dependencies are hosted on a private package index, set `indexUrl` to that index URL instead. Use `extraIndexUrls` to specify additional fallback indexes. ```yaml title="my_function.Function.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: my_function externalId: my_function indexUrl: https://@mycompany.com/pypi/simple extraIndexUrls: - https://:@pkgs.dev.azure.com//_packaging//pypi/simple/ - https://pypi.org/simple ``` Notice that for private package indexes, you often need to provide credentials to access the packages. Cognite Functions uses `pip` to install dependencies and `pip` supports basic authentication. For more details, see the [pip authentication documentation](https://pip.pypa.io/en/stable/topics/authentication/#basic-http-authentication). #### Include wheels An alternative to using a private package index is to build the wheel for the private package and include it in the function code directory. The Cognite Toolkit will upload the wheel file to CDF which will then install the package from the wheel file when installing the dependencies for the function. To build a wheel you need to use a package manager such as `pip`, `poetry`, or `uv`, and run the following command in the directory. ```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} pip wheel . ``` ```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} uv build ``` ```sh theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} poetry build ``` Your `requirements.txt` file should look like the following, where `my_private_package-0.1-py3-none-any.whl` is the wheel file for the private package that you want to include. ```text title="requirements.txt" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite-sdk function/my_private_package-0.1-py3-none-any.whl ``` Note that the `function/` prefix is a literal path required by CDF, not a placeholder. Replace `my_private_package-0.1-py3-none-any.whl` with your actual wheel filename. The function folder is expected to have the `my_private_package-0.1-py3-none-any.whl` file in the same directory. ```shell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} functions/ my_function.yaml schedules.yaml fn_example_repeater/ handler.py requirements.txt my_private_package-0.1-py3-none-any.whl ``` ### Function schedules **Resource directory: *functions/*** **API documentation:** [Schedules](/api-reference/concepts/20230101/function-schedules) Schedules for functions are also stored in the module's *functions/* directory. The Cognite Toolkit expects the YAML filename to end with `Schedule`, for example, *run\_calculation\_x.Schedule.yaml*. You can specify more than one schedule in a single file. To ensure that the function exists before the schedule is created, schedules are deployed after functions Schedules don't have `externalId`s, and the Cognite Toolkit identifies the schedule by a combination of the `functionExternalId` and the `name`. Consequently, **you can't deploy two schedules for a function with the exact same name**, and with two different sets of data. ```yaml title="daily.Schedule.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - name: 'daily-8am-utc' functionExternalId: 'fn_example_repeater' description: 'Run every day at 8am UTC' cronExpression: '0 8 * * *' data: breakfast: 'today: peanut butter sandwich and coffee' lunch: 'today: greek salad and water' dinner: 'today: steak and red wine' authentication: # Credentials to use to run the function in this schedule. # In this example, we just use the main deploy credentials, so the result is the same, but use a different set of # credentials (env variables) if you want to run the function with different permissions. clientId: { { myfunction_clientId } } clientSecret: { { myfunction_clientSecret } } - name: 'daily-8pm-utc' functionExternalId: 'fn_example_repeater' description: 'Run every day at 8pm UTC' cronExpression: '0 20 * * *' data: breakfast: 'tomorrow: peanut butter sandwich and coffee' lunch: 'tomorrow: greek salad and water' dinner: 'tomorrow: steak and red wine' ``` The `functionExternalId` must match an existing function or a function deployed by the tool. For schedules, the `authentication` property is optional but recommended. You can use it to specify credentials for the schedule that are different from the default credentials used by the Cognite Toolkit. We recommend using credentials with the **minimum** required access rights to run the function . If you don't specify the `authentication` property, the Cognite Toolkit uses its own credentials to run the function. This only works if the `config.[env].yaml` has set `validation-type: dev`. If it is set to `prod` or anything else, the Cognite Toolkit will raise an error when trying to deploy the schedule without the `authentication` property. ## Hosted extractors *Requires Cognite Toolkit v0.3.0 or later* **Resource directory: *hosted\_extractors/*** **Hosted extractor documentation:** [Hosted extractors](/cdf/integration/guides/extraction/hosted_extractors) The hosted extractors are stored in the module's `hosted_extractors/` directory. A hosted extractor has four types of resources: `Source`, `Destination`, `Job`, and `Mapping`. Each resource type has its suffix in the filename, for example, `my_kafka.Source.yaml`. When creating, updating, and deleting hosted extractors, the Cognite Toolkit applies changes in the correct order based on the dependencies between the source, destination, job, and mapping versions. ### Source **API documentation:** [Hosted extractor source](/api-reference/concepts/20230101/hosted-extractors-sources) Below is an example of a source configuration file. ```yaml title="my_mqtt.Source.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: mqtt5 externalId: my_mqtt host: mqtt.example.com port: 1883 authentication: username: myuser password: ${my_mqtt_password} ``` ### Destination **API documentation:** [Hosted extractor destination](/api-reference/concepts/20230101/hosted-extractors-destinations) Below is an example of a destination configuration file. ```yaml title="my_cdf.Destination.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_cdf credentials: clientId: ${my_cdf_clientId} clientSecret: ${my_cdf_clientSecret} targetDataSetExternalId: ds_files_hamburg ``` The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. ### Job **API documentation:** [Hosted extractor job](/api-reference/concepts/20230101/hosted-extractors-jobs) Below is an example of a job configuration file. ```yaml title="my_mqtt_to_cdf.Job.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_mqtt_to_cdf sourceId: my_mqtt destinationId: my_cdf format: type: value encoding: utf-16 compression: gzip ``` ### Mapping **API documentation:** [Hosted extractor mapping](/api-reference/concepts/20230101/hosted-extractors-mappings) Below is an example of a mapping configuration file. ```yaml title="my_mqtt_to_cdf.Mapping.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_mqtt_to_cdf mapping: expression: '[{ "type": "datapoint", "timestamp": to_unix_timestamp(input.timestamp, "%Y-%m-%dT%H:%M:%S"), "value": try_float(input.value, null), "externalId": input.tag }].filter(datapoint => datapoint.value is not null)' input: type: json published: true ``` For more information about the mapping configuration, see the [Hosted extractor documentation](/cdf/integration/guides/extraction/hosted_extractors/kuiper_concepts). ## Locations *Requires Cognite Toolkit v0.3.0 or later* **Resource directory: *locations/*** The location filters are stored in the module's *locations/* directory. You can have one or multiple locations in a single YAML file. The location YAML file name must end with `LocationFilter`, for example, `my.LocationFilter.yaml`. Location filters work with data modeling or with asset-centric resource types. The below example shows a location filter for data modeling. ```yaml title="example.LocationFilter.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: unique-external-id-123 name: 'Example location name' description: 'This is a description of the location.' parentExternalId: 'The parent location external ID' dataModels: - externalId: CogniteProcessIndustries space: cdf_idm version: v1 instanceSpaces: - instance-space-main - instance-space-secondary dataModelingType: DATA_MODELING_ONLY ``` Asset-centric location filters apply to assets, time series, events, sequences, and events. You can use a shared filter for all of these: ```yaml title="shared_filter.LocationFilter.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: unique-external-id-123 name: 'Example location name' parentId: 1 description: 'This is a description of the location.' assetCentric: dataSetExternalIds: - ds_data_set_890 assetSubtreeIds: - externalId: general-subtree-id-890 externalIdPrefix: general-prefix dataModelingType: HYBRID ``` It's common to use either `dataSetExternalId`, `assetSubtreeId`, or `externalIdPrefix` in the filter. The example below illustrates all the options. You can also set filters for specific resource types, such as `assets`, `events`, `files`, `timeseries`, and `sequences`: ```yaml title="shared_filter.LocationFilter.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: unique-external-id-123 name: 'Example location name' parentId: 1 description: 'This is a description of the location.' assetCentric: assets: dataSetExternalIds: - ds_data_set_123 assetSubtreeIds: - externalId: root-asset externalIdPrefix: asset-prefix events: dataSetExternalIds: - ds_data_set_456 assetSubtreeIds: - externalId: event-subtree-id-678 externalIdPrefix: event-prefix files: dataSetExternalIds: - ds_data_set_789 assetSubtreeIds: - externalId: file-subtree-id-901 externalIdPrefix: file-prefix timeseries: dataSetExternalIds: - ds_data_set_234 assetSubtreeIds: - externalId: timeseries-subtree-id-234 externalIdPrefix: timeseries-prefix sequences: dataSetExternalIds: - ds_data_set_567 assetSubtreeIds: - externalId: sequence-subtree-id-567 externalIdPrefix: sequence-prefix ``` The location filter API uses an **internal ID** for the `parentId` and `dataSetId`, while the YAML configuration files reference the **external ID**; `parentExternalid` and `dataSetExternalId`. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the Cognite API. ## RAW **Resource directory: *raw/*** **API documentation:** [RAW](/api-reference/concepts/20230101/raw) The RAW configuration files are stored in the module's *raw/* directory. You can have one or more RAW configurations in a single YAML file. For example, multiple tables can be defined in a single file. RAW table configuration filenames must end with `Table`, for example, `my_table.Table.yaml`. Database configuration filenames must end with `Database`, for example, `my_database.Database.yaml`. ```yaml title="raw_tables.Table.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - dbName: sap tableName: workorder_mdi2_sap - dbName: sap tableName: workorxder_mdi2_sap2 ``` Or you can define one table per file. ```yaml title="sap_workorder_mdi2_sap.Table.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} dbName: sap tableName: workorder_mdi2_sap ``` ### Uploading data to RAW tables Use the Cognite Toolkit only to upload **example data**, and not as a general solution to ingest data into CDF. However, there are use cases where uploading data to RAW tables can be useful see [Use case: Uploading data to RAW tables](#use-case-uploading-data-to-raw-tables). You can upload data to RAW tables. You need to create one YAML file per table you want to upload. The data file can either be a *.csv* or *.parquet* file and must be named the same name as the YAML file. This example configuration creates a RAW database called `asset_hamburg_sap` with a table called `assets` and populates it with data from the *asset\_hamburg\_sap.csv* file. ```yaml title="asset_hamburg_sap.Table.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} dbName: asset_hamburg_sap tableName: assets ``` ```csv title="asset_hamburg_sap.Table.csv" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} "key","categoryId","sourceDb","parentExternalId","updatedDate","createdDate","externalId","isCriticalLine","description","tag","areaId","isActive" "WMT:48-PAHH-96960","1152","workmate","WMT:48-PT-96960","2015-10-06 12:28:33","2013-05-16 11:50:16","WMT:48-PAHH-96960","false","VRD - PH STG1 COMP WTR MIST RELEASED : PRESSURE ALARM HIGH HIGH","48-PAHH-96960","1004","true" "WMT:48-XV-96960-02","1113","workmate","WMT:48-XV-96960","2015-10-08 08:48:04","2009-06-26 15:36:40","WMT:48-XV-96960-02","false","VRD - PH STG1 COMP WTR MIST WTR RLS","48-XV-96960-02","1004","true" "WMT:23-TAL-96183","1152","workmate","WMT:23-TT-96183","2015-10-06 12:28:32","2013-05-16 11:50:16","WMT:23-TAL-96183","false","VRD - PH 1STSTG COMP OIL TANK HEATER : TEMPERATURE ALARM LOW","23-TAL-96183","1004","true" ``` If the leftmost column in the CSV file is named `key`, the Cognite Toolkit will use this column as the index column for the table. #### Use case: Uploading data to RAW tables The Cognite Toolkit governs resource configurations, typically **metadata** rather than **data**. For example, a sensor's name, location, type, and the asset it's attached to are metadata, while the actual sensor readings are data. Metadata is typically available from a source system. You can, for example, use an extraction pipeline to extract and ingest the metadata to CDF. If the metadata isn't available for extraction from a source system, a potential option is to store the metadata as .csv files and have them version-controlled, for example, in a Git repository. Next, you can use the Cognite Toolkit to deploy the metadata to RAW tables in CDF. Then, you can use Transformations to write the metadata to the correct destination resources. This way, you can track changes to the metadata and use the Git repository as the single source of truth for the metadata. ## Relationships *Requires Cognite Toolkit v0.4.0 or later* **Resource directory: *classic/*** **API documentation:** [Relationships](/api-reference/concepts/20230101/relationships) Relationships can be found in the module's `classic/` directory. You can define one or more relationships in a single YAML file. The filename must end with `Relationship`, for example, `my_relationship.Relationship.yaml`. The Cognite Toolkit ensures that dependent resources are created before the relationships. For example, if you reference an asset or a data set in a relationship, the Cognite Toolkit creates the asset or data set before creating the relationship. Example relationship definition: ```yaml title="my_relationship.Relationship.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: MyRelationship sourceType: asset sourceExternalId: MyAsset targetType: event targetExternalId: MyEvent dataSetExternalId: ds_complete_org confidence: 0.42 startTime: 1732959346052 endTime: 1732959346052 labels: - externalId: my_label - externalId: my_other_label ``` The data set is referenced by the `dataSetExternalId`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. ## Resource view mapping *Requires Cognite Toolkit v0.6.73 or later and the alpha flag `migration` enabled* **Resource directory: *migration/*** `ResourceViewMapping` is not an API. It is a special resource that is used by the migration tooling available as an alpha feature in the Cognite Toolkit. It requires the migration model to be present in your CDF project. `ResourceViewMapping` can be found in the modules `migration/` directory. You can define one or more `ResourceViewMapping` in a single YAML file. The filename must end with `ResourceViewMapping`, for example, `my_asset_mapping.ResourceViewMapping.yaml`. Example ResourceViewMapping: ```yaml title="my_asset_mapping.ResourceViewMapping.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_asset_mapping # Unique identifier for the mapping resourceType: asset # The resource type being mapped, e.g., asset, timeseries, event, file, sequence viewId: # The ID of the ingestion view space: cdf_cdm externalId: CogniteAsset version: v1 propertyMapping: # The mapping from asset-centric properties to properties in the ingestion view # Expects json path for the asset-centric property values. name: name description: description labels: tags parentId: parent metadata.altName: alias ``` ## Robotics *Requires Cognite Toolkit v0.3.0 or later* **Resource directory: *robotics/*** **API documentation:** The Robotics API is not yet publicly available. The Robotics configuration files are stored in the module's *robotics/* directory. There are multiple types of Robotics resources: `RobotCapability`, `Map`, `Location`, `Frame`, `DataPostProcessing`. You can have one or more resources in a single YAML file, but all resources in the file must be of the same type. Each resource type has its suffix in the filename, for example, `my_robot_capability.RobotCapability.yaml`. ### Robot capabilities Below is an example of a RobotCapability configuration file. ```yaml title="my_robot_capability.RobotCapability.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: ptz externalId: ptz method: ptz description: Description of the PTZ camera capability inputSchema: $schema: http://json-schema.org/draft-07/schema# id: robotics/schemas/0.1.0/capabilities/ptz title: PTZ camera capability input type: object properties: method: type: string parameters: type: object properties: tilt: type: number minimum: -90 maximum: 90 pan: type: number minimum: -180 maximum: 180 zoom: type: number minimum: 0 maximum: 100 required: - tilt - pan - zoom required: - method - parameters additionalProperties: false dataHandlingSchema: $schema: http://json-schema.org/draft-07/schema# id: robotics/schemas/0.1.0/data_handling/ptz type: object properties: uploadInstructions: type: object properties: image: type: object properties: method: const: uploadFile parameters: type: object properties: filenamePrefix: type: string required: - filenamePrefix required: - method - parameters additionalProperties: false additionalProperties: false required: - uploadInstructions ``` In the above schema, we have: * **Required properties**: `name`, `externalId` and `method`. * **Optional properties**: `description`, `inputSchema`, `dataHandlingSchema`. * `inputSchema` and `dataHandlingSchema` are objects and are not verified by the Cognite Toolkit, they are passed as is to the Robotics API. ### Map Below is an example of a Map configuration file. ```yaml title="my_map.Map.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: Robot navigation map externalId: robotMap mapType: POINTCLOUD description: A map of the robot's navigation environment frameExternalId: robotFrame data: filename: map.ply mimeType: application/octet-stream locationExternalId: robotLocation scale: 1.0 ``` In the above schema, we have: * **Required properties**: `name`, `externalId`, and `mapType`. * **Optional properties**: `description`, `data`, `locationExternalId`, `scale`. * `MapType` has allowed values `WAYPOINTMAP`, `THREEDMODEL`, `TWODMAP`, and `POINTCLOUD` * `data` is an object that is not verified by the Cognite Toolkit, it is passed as is to the Robotics API. ### Location Below is an example of a Location configuration file. ```yaml title="my_location.Location.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: Water treatment plant externalId: waterTreatmentPlant1_Windows_3_11_8 description: Original Description ``` In the above schema, we have: * **Required properties**: `name` and `externalId`. * **Optional properties**: `description`. ### Frame Below is an example of a Frame configuration file. ```yaml title="my_frame.Frame.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: Some coordinate frame externalId: someCoordinateFrame transform: parentFrameExternalId: rootCoordinateFrame translation: x: 0 y: 0 z: 0 orientation: x: 0 y: 0 z: 0 w: 1 ``` In the above schema, we have: * **Required properties**: `name` and `externalId`. * **Optional properties**: `transform`. * In `transform`, * **Required properties**: `parentFrameExternalId`, `translation`, `orientation`. * **Optional properties**: None. * For `translation` and `orientation`, all properties are required. ### Data post-processing Below is an example of a DataPostProcessing configuration file. ```yaml title="my_data_post_processing.DataPostProcessing.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} name: Read dial gauge externalId: read_dial_gauge method: read_dial_gauge description: Original Description inputSchema: $schema: http://json-schema.org/draft-07/schema# id: robotics/schemas/0.1.0/capabilities/ptz title: PTZ camera capability input type: object properties: method: type: string parameters: type: object properties: tilt: type: number minimum: -90 maximum: 90 pan: type: number minimum: -180 maximum: 180 zoom: type: number minimum: 0 maximum: 100 required: - tilt - pan - zoom required: - method - parameters additionalProperties: false ``` In the above schema, we have: * **Required properties**: `name`, `externalId`, and `method`. * **Optional properties**: `description`, `inputSchema`. * `inputSchema` is an object and is not verified by the Cognite Toolkit, it is passed as is to the Robotics API. ## Search configuration *Requires Cognite Toolkit v0.6.26 or later and alpha flag `search-config` enabled* **Resource directory: *cdf\_applications/*** SearchConfig can be found in the modules `cdf_applications/` directory. You can define search configurations for a view in a single YAML file. The filename must end with `SearchConfig`, for example, `MyActivity_view.SearchConfig.yaml`. The Cognite Toolkit accepts a single SearchConfig YAML for a view where you can define the configurations that you want to see in your search applications. Example SearchConfig configuration: ```yaml title="MyActivity_view.SearchConfig.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} view: externalId: MyActivity space: sp_activity_model useAsName: MyActivity Search Configuration useAsDescription: MyActivity view's search application configurations columnsLayout: - property: name selected: true hidden: false - property: description selected: false hidden: true - property: externalId selected: true hidden: false filterLayout: - property: name selected: true hidden: false - property: description selected: false hidden: true - property: externalId selected: true hidden: false propertiesLayout: - property: name selected: true hidden: false - property: description selected: false hidden: true - property: externalId selected: true hidden: false ``` Only one YAML resource can be created for a view. If multiple resources are created, the toolkit will skip other resources with warning. It is prefered to list down all the properties under `columnsLayout`, `filterLayout` and `propertiesLayout`. Mention the properties, filters, and columns in the order you want them to appear in the search application. ## Sequences *Requires Cognite Toolkit v0.3.0 or later* Sequence rows requires Cognite Toolkit v0.4.0 or later **Resource directory: *classic/*** **API documentation:** [Sequences](/api-reference/concepts/20230101/sequences) **API documentation Rows:** [Sequence Rows](/api-reference/concepts/20230101/sequences) Sequences can be found in the module's `classic/` directory. You can define one or more sequences in a single YAML file. The filename must end with `Sequence`, for example, `my_sequence.Sequence.yaml`. Below is an example of a sequence configuration file. ```yaml title="my_sequence.Sequence.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: windturbine_powercurve_xyz name: Wind turbine power curve XYZ description: A power curve for a wind turbine model XYZ dataSetExternalId: ds_sequences assetExternalId: wind_turbine_xyz metadata: manufacturer: WindTurbineCorp model: XYZ columns: - externalId: wind_speed valueType: DOUBLE description: Wind speed in m/s name: Display name for wind speed - externalId: power valueType: DOUBLE description: Power in kW name: Display name for power ``` The data set is referenced by the `dataSetExternalId`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. ### Sequence rows Sequence **rows** are a separate resource type from sequences. The sequence row configuration files are expected in the *classic/* directory and must have the suffix `SequenceRow`, for example, `my_sequence.SequenceRow.yaml`. Below is an example of a sequence row configuration file. ```yaml title="powercurve.SequenceRow.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: SequenceWithRows columns: - wind_speed - power rows: - rowNumber: 1 values: - 0.0 - 0.0 - rowNumber: 2 values: - 5.0 - 309000.0 - rowNumber: 3 values: - 10.0 - 2693000.0 - rowNumber: 4 values: - 15.0 - 3450000.0 - rowNumber: 5 values: - 20.0 - 3450000.0 - rowNumber: 6 values: - 25.0 - 3450000.0 ``` The `externalId` is the same as the sequence's `externalId`. The `columns` property lists the columns in the sequence. The `rows` property contains the data for the sequence. Each row has a `rowNumber` and a `values` property. The `values` property is an array of values for each column in the sequence. ## Signals *Requires Cognite Toolkit v0.7.166 or later and alpha flag `signals` enabled* Enable the alpha flag by setting `signals = true` under `[alpha_flags]` in your `cdf.toml`. **Resource directory: *signals/*** The `signals/` directory holds **signal sinks** (delivery targets for signals) and **signal subscriptions** (filters that route matching signals to sinks). The YAML kinds are `Sink` and `Subscription`. ### Signal sinks **API documentation:** [Signals](/api-reference/concepts/20230101-alpha/signals) (signal sinks) Signal sinks define where signals are delivered. Store one or more sinks per YAML file. The filename must end with `.Sink`, for example, `alerts.Sink.yaml`. Example signal sink (webhook) configuration: ```yaml title="alerts.Sink.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: alert_email_sink type: email emailAddress: alerts@example.com - externalId: alternative_email_sink type: email emailAddress: alternative_alerts@example.com ``` Signals are an alpha feature. Schemas and available sink types may change. The Cognite Toolkit forwards the `configuration` object as-is and performs syntax validation only. ### Signal subscriptions **API documentation:** [Signals](/api-reference/concepts/20230101-alpha/signals) (signal subscriptions) Signal subscriptions select which signals to receive and which sinks to deliver them to. We recommend one subscription per file. The filename must end with `.Subscription`, for example, `high_temperature.Subscription.yaml`. Example signal subscription configuration: ```yaml title="high_temperature.Subscription.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: workflows-sub sink: type: user externalId: my-user-sink filter: topic: cognite_workflows resource: my-workflow category: [workflowExecutionFailed, workflowExecutionTimedOut, triggerAttemptFailed] severity: error ``` Signals are an alpha feature. Filter semantics and fields may change. The Cognite Toolkit forwards the `filter` object as-is and performs syntax validation only. ## Rule sets *Requires Cognite Toolkit v0.7.207 or later and alpha flag `data-products` enabled* Set `data-products = true` under `[alpha_flags]` in your `cdf.toml` to enable the alpha flag. **Resource directory: *rulesets/*** The `rulesets/` directory holds **rule sets** (parent resources) and **rule set versions** (immutable SHACL rules per version). The YAML kinds are `RuleSet` and `RuleSetVersion`. ### RuleSet A rule set is the parent container for immutable rule set versions. The filename must end with `.RuleSet`, for example, `my.RuleSet.yaml`. Example rule set configuration (from the [Cognite Toolkit](https://github.com/cognitedata/toolkit) test data): ```yaml title="my.RuleSet.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my-rule-set name: My Rule Set ``` Rule sets are an alpha feature. The API and YAML structure may change. The Cognite Toolkit does not support updating rule sets. To apply changes, delete the rule set and redeploy. ### RuleSetVersion Rule set versions hold SHACL rules in Turtle format and reference their parent rule set with `ruleSetExternalId`. The filename must end with `.RuleSetVersion`, for example, `my.RuleSetVersion.yaml`. Either add Turtle under `rules` as a list of strings (each string is one Turtle document), or omit `rules` and put the Turtle in a `.ttl` file in the same directory as the YAML file. Name the file `{filename stem}.ttl` or `{ruleSetExternalId}.ttl`. When you use a `.ttl` file, the Cognite Toolkit reads it at build time and sends its contents as `rules`. Rule set versions are uniquely identified by their `ruleSetExternalId` and `version`. `version` must follow the semantic versioning format (for example, `1.0.0`). Example rule set version configuration: ```yaml title="my.RuleSetVersion.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ruleSetExternalId: my-rule-set version: "1.0.0" rules: - | @prefix sh: . @prefix ex: . ex:MyShape a sh:NodeShape . ``` Rule set versions are immutable. The Cognite Toolkit does not support updating rule set versions in place. SHACL rule semantics and the API may change. The Cognite Toolkit forwards rule content as-is and validates syntax only. ## Simulator models *Requires Cognite Toolkit v0.8.0 or later. In earlier versions, enable the `simulators` alpha flag.* **Resource directory: *simulators/*** **API documentation:** [Simulator Models](/api-reference/concepts/20230101/simulator-models) Simulator models are stored in the module's `simulators/` directory. The filename must end with `SimulatorModel`, for example, `my_simulator_model.SimulatorModel.yaml`. Below is an example of a simulator model configuration file. ```yaml title="my_simulator_model.SimulatorModel.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_simulator_model name: My Simulator Model description: This is an example simulator model for demonstration purposes. type: SteadyState dataSetExternalId: my_simulator_data_set simulatorExternalId: my_simulator ``` ## Simulator model revisions *Requires Cognite Toolkit v0.8.0 or later. In earlier versions, enable the `simulators` alpha flag.* **Resource directory: *simulators/*** **API documentation:** [Simulator Model Revisions](/api-reference/concepts/20230101/simulator-models) Simulator model revisions are stored in the module's `simulators/` directory. The filename must end with `SimulatorModelRevision`, for example, `my_simulator_model_revision.SimulatorModelRevision.yaml`. Below is an example of a simulator model revision configuration file. ```yaml title="my_simulator_model_revision.SimulatorModelRevision.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_simulator_model_revision modelExternalId: my_simulator_model description: This is an example simulator model revision for demonstration purposes. fileExternalId: my_simulator_model_file ``` The Toolkit automatically resolves the `fileExternalId` to the internal ID of the file. ## Simulator routine *Requires Cognite Toolkit v0.8.0 or later. In earlier versions, enable the `simulators` alpha flag.* **Resource directory: *simulators/*** **API documentation:** [Simulator Routines](/api-reference/concepts/20230101/simulator-routines) Simulator routines are stored in the module's `simulators/` directory. The filename must end with `SimulatorRoutine`, for example, `my_simulator_routine.SimulatorRoutine.yaml`. Below is an example of a simulator routine configuration file. ```yaml title="my_simulator_routine.SimulatorRoutine.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_simulator_routine modelExternalId: my_simulator_model simulatorIntegrationExternalId: my_simulator_integration name: My Simulator Routine description: This is an example simulator routine for demonstration purposes. ``` ## Simulator routine revision *Requires Cognite Toolkit v0.8.0 or later. In earlier versions, enable the `simulators` alpha flag.* **Resource directory: *simulators/*** **API documentation:** [Simulator Routine Revisions](/api-reference/concepts/20230101/simulator-routines) Simulator routine revisions are stored in the module's `simulators/` directory. The filename must end with `SimulatorRoutineRevision`, for example, `my_simulator_routine_revision.SimulatorRoutineRevision.yaml`. Below is an example of a simulator routine revision configuration file. ```yaml title="my_simulator_routine_revision.SimulatorRoutineRevision.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_simulator_routine_revision routineExternalId: my_simulator_routine configuration: schedule: enabled: false dataSampling: enabled: true samplingWindow: 15 granularity: 1 validationWindow: 5 logicalCheck: - enabled: true timeseriesExternalId: simulator_routine_revision_logical_check aggregate: average operator: lt value: 75.8 steadyStateDetection: - enabled: true aggregate: average minSectionSize: 1 varThreshold: 3.5 slopeThreshold: 0.5 inputs: - name: Cold water Temperature referenceId: CWT value: 10.0 valueType: DOUBLE unit: name: C quantity: temperature - name: Cold Water Pressure referenceId: CWP value: - 3.6 valueType: DOUBLE_ARRAY unit: name: bar quantity: pressure outputs: - name: Shower Temperature referenceId: ST unit: name: C quantity: temperature valueType: DOUBLE script: - order: 1 description: Set Inputs steps: - order: 1 stepType: Set description: Set Cold water Temperature arguments: referenceId: CWT objectName: Cold Water objectProperty: Temperature - order: 2 stepType: Set description: Set Cold Water Pressure arguments: referenceId: CWP objectName: Cold Water objectProperty: Pressure - order: 2 description: Solve the flowsheet steps: - order: 1 stepType: Command arguments: command: Solve - order: 3 description: Set simulator outputs steps: - order: 1 stepType: Get arguments: referenceId: ST objectName: Shower objectProperty: Temperature ``` ## Custom apps **Resource directory: *apps/*** *Requires Cognite Toolkit v0.8.82 or later and alpha flag `custom-apps` enabled* **API documentation:** [Custom apps](/cdf/explore/custom_apps) Alpha flags enable experimental Cognite Toolkit behavior that may change or be removed without notice. Cognite Flows custom apps are web applications deployed to CDF via the app-hosting API. The Cognite Toolkit handles packaging and uploading the built app as an alternative to using `npx @cognite/cli@latest apps deploy`. For the full CLI-based workflow, see [Deploying Flows custom apps](/cdf/flows/guides/deploying). Custom app configurations are stored in the module's *apps/* directory. A custom app has two resource types: `App` (identity and metadata) and `AppVersion` (versioned deployment with source code upload). They are identified by the `.App.yaml` and `.AppVersion.yaml` filename suffixes. ### Source code requirements The app source must be a standard Node.js project. These files must exist at the **app root** (the project source directory, not in `dist/`): * `package.json` * `package-lock.json` Build the app before you run `cdf build` (typically `npm run build`). That step creates a `dist/` directory in the app source folder. Don't modify files in `dist/` manually. If `dist/` is missing, the Cognite Toolkit prompts you to run the build command first. ### Directory structure **Option 1: (recommended)** Source code in a root-level `apps/` folder (use `sourcePath`) The recommended layout for monorepos is to keep the JS app at the repository root, with toolkit modules in a `modules/` folder alongside it. Here is an example structure: ```shell wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} apps/ # root-level app source, sibling to modules/ my-custom-app/ dist/ # auto-generated build output from vite — must exist before running cdf build index.html ... package.json package-lock.json manifest.json src/ ... modules/ / apps/ my-app.App.yaml my-app.AppVersion.yaml # sourcePath: ../../../apps/my-custom-app ``` **Option 2:** Source code in the module `apps/` folder (omit `sourcePath`) Place the app source in the same `apps/` directory as the `.App.yaml` and `.AppVersion.yaml` files. When the folder name matches `appExternalId`, you can omit `sourcePath`. ```shell wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} modules/ / apps/ my-app.App.yaml my-app.AppVersion.yaml my-custom-app/ # named after appExternalId dist/ # auto-generated build output from vite — must exist before running cdf build index.html ... package.json package-lock.json manifest.json ... ``` Set `sourcePath` in the app version `.AppVersion.yaml` file to the relative path from the YAML file to the app root: ```yaml title="my-app.AppVersion.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} appExternalId: my-app version: 1.0.0 lifecycleState: PUBLISHED sourcePath: ../../../apps/my-custom-app ``` `sourcePath` is relative to the directory that contains `.AppVersion.yaml`. Prefer **Option 1** or **Option 2** instead of ad hoc paths elsewhere on disk. ### App configuration The `App` resource defines the stable identity and display metadata for the application. ```yaml title="my-app.App.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my-app name: My Custom App description: Optional description of the app ``` ### App version configuration The `AppVersion` resource defines a specific version of the app and controls its lifecycle and visibility. An app can have multiple versions at the same time. ```yaml title="my-app.AppVersion.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} appExternalId: my-app version: 1.0.0 lifecycleState: PUBLISHED # DRAFT | PUBLISHED | DEPRECATED | ARCHIVED alias: ACTIVE # ACTIVE | PREVIEW — optional, at most one per app entrypoint: index.html # defaults to index.html sourcePath: ../my-custom-app # optional; defaults to sibling directory alongside the YAML file named after appExternalId ``` The `lifecycleState` values map to the [Flows custom app lifecycle](/cdf/flows/concepts/app-lifecycle): | Value | Description | | ------------ | --------------------------------------------------------- | | `DRAFT` | Deployed but not yet accessible to end users. | | `PUBLISHED` | Accessible to users; version is immutable once published. | | `DEPRECATED` | Still accessible, but signaled as outdated. | | `ARCHIVED` | No longer accessible to users | The `alias` field controls which version users see from **Custom apps** or a preview URL: | Value | Description | | --------- | -------------------------------------------------- | | `ACTIVE` | This version is served at the app's canonical URL. | | `PREVIEW` | This version is served at a preview URL. | Each alias type has its own constraints: * **One `ACTIVE` per app** — At most one version per app can have `alias: ACTIVE`. This is the [active version](/cdf/flows/concepts/app-lifecycle) users open from **Custom apps**. * **`PUBLISHED` only** — Set `alias: ACTIVE` only when `lifecycleState` is `PUBLISHED`. * **Automatic transfer** — When you assign `ACTIVE` to a version, Cognite Toolkit removes `ACTIVE` from the version that had it before. * **One `PREVIEW` per app** — At most one version per app can have `alias: PREVIEW`. * **`DEPRECATED` does not clear `ACTIVE`** — If you change an `ACTIVE` version to `DEPRECATED`, it keeps `ACTIVE` and remains served at the canonical URL until you clear the alias. * **Clear an alias** — Set `alias: null` in that version's YAML to remove `ACTIVE` or `PREVIEW` without assigning it to another version. ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} alias: null # removes ACTIVE or PREVIEW from this version ``` Deprecating a version does not remove the `ACTIVE` alias. Users still open that build from **Custom apps** until you set `alias: null` or assign `ACTIVE` to another published version. ### Deploy and update behavior * **`App`** — Cognite Toolkit does not update `name` or `description` after the first deploy. Set `externalId`, `name`, and `description` in `.App.yaml` before the initial `cdf deploy`. * **`AppVersion`** — To update an existing version, edit its `.AppVersion.yaml` (and rebuild the app if source files changed), then run `cdf build` and `cdf deploy` with the same `version` value. Cognite Toolkit re-uploads the zip from `dist/` and applies changes to `lifecycleState`, `alias`, `entrypoint`, and `sourcePath`. You cannot revert `lifecycleState` from `PUBLISHED` to `DRAFT`. To ship a parallel release, add another `.AppVersion.yaml` with a new `version` value. ### Further reading * [App lifecycle](/cdf/flows/concepts/app-lifecycle) — Draft, Published, and active version concepts. * [Deploying Flows custom apps](/cdf/flows/guides/deploying) — CLI-based deployment workflow. ## Streamlit applications *Requires Cognite Toolkit v0.4.0 or later* **Resource directory: *streamlit/*** **API documentation:** This uses the files API: [Files](/api-reference/concepts/20230101/file-content) Streamlit applications are stored in the module's *streamlit/* directory. You can define one or more Streamlit applications in a single YAML file. The filename must end with `Streamlit`, for example, `myapp.Streamlit.yaml`. Below is an example of a Streamlit application configuration file. ```yaml title="myapp.Streamlit.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: myapp name: MySuperApp creator: doctrino@github.com description: This is a super app published: true theme: Light thumbnail: 'data:image/webp;base64,....' dataSetExternalId: ds_complete_org entrypoint: main.py ``` The data set is referenced by the `dataSetExternalId`. The Cognite Toolkit automatically resolves the external ID to the internal ID of the data set. The `externalId` of the application must be unique within the project and must match the name of a directory where the `.py` files are located, including the `entrypoint` file. In addition, there must be a `requirements.txt` file in the same directory. For the above example, the directory structure would look like this: ```shell theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} / streamlit/ myapp.Streamlit.yaml myapp/ main.py requirements.txt ``` ## Streams *Requires Cognite Toolkit v0.8.0 or later. In earlier versions, enable the `streams` alpha flag.* **Resource directory: *streams/*** **API documentation:** [Streams](/api-reference/concepts/20230101/streams) Streams are stored in the modules' `streams/` directory. The filename must end with `Streams`, for example, `my_stream.Streams.yaml`. The number of active streams has limits that depend on the template name. For more information about these limits, see the API documentation. Example stream configuration: ```yaml title="my_stream.Streams.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my-stream settings: template: name: ImmutableTestStream ``` Currently, three template types are available for the `name` field: `ImmutableTestStream`, `BasicArchive`, and `BasicLiveData`. ## Transformations **Resource directory: *transformations/*** Transformation Notifications requires Cognite Toolkit v0.3.0 or later **API documentation Transformations:** [Transformations](/api-reference/concepts/20230101/transformations) **API documentation Schedules:** [Schedule](/api-reference/concepts/20230101/transformation-schedules) **API documentation Notifications:** [Notifications](/api-reference/concepts/20230101/transformation-notifications) The transformation configuration files are stored in the module's *transformations/* directory. You can have one or more transformations in a single YAML file, but typically you have one transformation per file. The filename must end with `Transformation`, for example, `my_transformation.Transformation.yaml`. Each transformation can have a corresponding *.sql* file with the accompanying SQL code. The *.sql* file should have the same filename as the YAML file that defines the transformation (without the number prefix) or use the *externalId* of the transformation as the filename. The transformation schedule is a separate resource type, tied to the transformation by `external_id`. The Cognite Toolkit detects the transformation schedule YAML file by the `schedule` suffix in the filename, for example, `my_transformation.schedule.yaml`. The transformation notification YAML file is detected by the `Notification` suffix in the filename, for example, `my_transformation.Notification.yaml`. All other YAML files are considered transformation configurations. ### Transformation configuration **Example transformation configuration:** ```yaml title="tr_asset_oid_workmate_asset_hierarchy.Transformation.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'tr_asset_{{location_name}}_{{source_name}}_asset_hierarchy' dataSetExternalId: 'ds_asset_{{location_name}}' name: 'asset:{{location_name}}:{{source_name}}:asset_hierarchy' destination: type: 'asset_hierarchy' ignoreNullFields: true isPublic: true conflictMode: upsert tags: - 'asset_hierarchy' # Specify credentials separately like this: # You can also use different credentials for running the transformations than the credentials you use to deploy. authentication: clientId: { { cicd_clientId } } clientSecret: { { cicd_clientSecret } } tokenUri: { { cicd_tokenUri } } # Optional: If idP requires providing the cicd_scopes cdfProjectName: { { cdfProjectName } } scopes: { { cicd_scopes } } # Optional: If idP requires providing the cicd_audience audience: { { cicd_audience } } ``` ```sql title="tr_asset_oid_workmate_asset_hierarchy.Transformation.sql" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} SELECT externalId as externalId, if(parentExternalId is null, '', parentExternalId) as parentExternalId, tag as name, sourceDb as source, description, dataset_id('{{asset_dataset}}') as dataSetId, to_metadata_except( array("sourceDb", "parentExternalId", "description"), *) as metadata FROM `{{asset_raw_input_db}}`.`{{asset_raw_input_table}}` ``` You can configure the transformation with both **from** and **to** sets of credentials. Use the `authentication` section to specify the credentials. See below for more information on how to configure the credentials. You can specify the SQL inline in the transformation YAML file, using the `query` property (str), but we recommend that you use a separate *.sql* file for readability. In the above transformation, the transformation re-uses the globally defined credentials for the Cognite Toolkit. For production projects, we recommend that you use a service account with the **minimum** required access rights instead. ### Transformation credentials You can specify the credentials in the `authentication` section of the transformation YAML file. The most common way is as follows: ```yaml title= theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ... authentication: clientId: my_client_id clientSecret: my_client_secret ``` Here, `clientId` and `clientSecret` are used to authenticate against a service principal that is connected to a Cognite group with the capabilities (authorization) needed to run the transformation. The service principal must be authorized to read from the source table and write to the transformation destination, for example, a data model. You can also specify two sets of credentials, one for reading from the source and one for writing to the destination. ```yaml title= theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} ... authentication: read: clientId: read_service_principle_client_id clientSecret: read_service_principle_client_secret write: clientId: destination_service_principle_client_id clientSecret: destination_service_principle_client_secret ``` In this case, the `read` credentials are used to read from the source table, and the `write` credentials are used to write to the destination. If you need to authenticate against a different CDF project, you can specify full credentials: ```yaml title= theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} authentication: clientId: my_client_id clientSecret: my_client_secret tokenUri: https://cognite.com/oauth/token cdfProjectName: my_cdf_project_name scopes: ['https://cognite.com/.default'] audience: 'https://cognite.com' ``` or full credentials for both reading and writing: ```yaml title= theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} authentication: read: clientId: read_service_principle_client_id clientSecret: read_service_principle_client_secret tokenUri: https://cognite.com/oauth/token cdfProjectName: my_cdf_project_name scopes: ['https://cognite.com/.default'] audience: 'https://cognite.com' write: clientId: destination_service_principle_client_id clientSecret: destination_service_principle_client_secret tokenUri: https://cognite.com/oauth/token cdfProjectName: my_cdf_project_name scopes: ['https://cognite.com/.default'] audience: 'https://cognite.com' ``` The most common use case for this is to read from one CDF project and write to another CDF project. ### Transformation schedule The transformation `schedule` is optional. If you do not specify a schedule, the transformation will be created, but not scheduled. You can then schedule it manually in the CDF UI or using the Cognite API. Schedule is a separate API endpoint in CDF. **Example transformation schedule configuration:** ```yaml title="tr_asset_oid_workmate_asset_hierarchy.schedule.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: 'tr_asset_{{location_name}}_{{source_name}}_asset_hierarchy' interval: '{{scheduleHourly}}' isPaused: {{ pause_transformations }} ``` ### Transformation notifications The transformation `notification` is optional. Below is an example of a transformation notification configuration file. ```yaml title="tr_notifications.Notification.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - transformationExternalId: tr_first_transformation destination: john.smith@example.com - transformationExternalId: tr_first_transformation destination: jane.smith@example.com ``` CDF identifies notifications by their **internal ID** while the Cognite Toolkit uses a combination of the **transformation external ID** and the **destination** to identify each notification Running `cdf clean` deletes **all** notifications for a transformation external ID and destination. ## Time series **Resource directory: *timeseries/*** **API documentation:** [Time-series](/api-reference/concepts/20230101/time-series) Use the Cognite Toolkit only to upload **example data**, and not as a general solution to ingest time series into CDF. TimeSeries can be found in the module's `timeseries/` directory. You can define the metadata of one or more time series in a single or multiple YAML file(s). The filename must end with `TimeSeries`, for example, `my_timeseries.TimeSeries.yaml`. Typically, you create time series when ingesting data into CDF by configuring the data pipelines with the corresponding data sets, databases, groups, and so on. Example time series configuration: ```yaml title="timeseries.TimeSeries.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} - externalId: 'pi_160696' name: 'VAL_23-PT-92504:X.Value' dataSetExternalId: ds_timeseries_hamburg isString: false metadata: compdev: '0' location5: '2' pointtype: Float32 convers: '1' descriptor: PH 1stStgSuctCool Gas Out contextMatchString: 23-PT-92504 contextClass: VAL digitalset: '' zero: '0' filtercode: '0' compdevpercent: '0' compressing: '0' tag: 'VAL_23-PT-92504:X.Value' isStep: false description: PH 1stStgSuctCool Gas Out assetExternalId: '23-PT-92504' - externalId: 'pi_160702' name: 'VAL_23-PT-92536:X.Value' dataSetExternalId: ds_timeseries_hamburg isString: false metadata: compdev: '0' location5: '2' pointtype: Float32 convers: '1' descriptor: PH 1stStgComp Discharge contextMatchString: 23-PT-92536 contextClass: VAL digitalset: '' zero: '0' filtercode: '0' compdevpercent: '0' compressing: '0' tag: 'VAL_23-PT-92536:X.Value' ``` This configuration creates two timeseries in the `ds_timeseries_hamburg` data set with the external IDs `pi_160696` and `pi_160702`. ### Uploading datapoints to time series Use the Cognite Toolkit only to upload **example data**, and not as a general solution to ingest time series into CDF. You can upload datapoints to times series using the Cognite Toolkit. The datapoints are stored in the module's *timeseries/* directory. Datapoints are stored in `csv` or `parquet` files. There is no requirements for the filename of the datapoints file. Typically, you create time series when ingesting data into CDF by configuring the data pipelines with the corresponding data sets, databases, groups, and so on. Example of datapoints: ```csv title="datapoints.csv" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} timestamp,pi_160696,pi_160702 2013-01-01 00:00:00,0.9430412044195982,0.9212588490581821 2013-01-01 01:00:00,0.9411303320132799,0.9212528389403117 2013-01-01 02:00:00,0.9394743147709556,0.9212779911470234 2013-01-01 03:00:00,0.9375842300608798, 2013-01-01 04:00:00,0.9355836846172971,0.9153202184209938 ``` This *.csv* file loads data into the time series created in the previous example. The first column is the timestamp, and the following columns are the external ID for the time series at that timestamp. ## Timeseries subscriptions *Requires Cognite Toolkit v0.2.0 or later* **Resource directory: *timeseries*/** **API documentation:** [Timeseries subscriptions](/api-reference/concepts/20230101/data-point-subscriptions) Timeseries subscriptions are stored in the module's *timeseries/* directory. We recommend to have a separate YAML file for each subscription. Use the `DatapointSubscription` suffix in the filename, for example `my_subscription.DatapointSubscription.yaml`. The Cognite Toolkit create the timeseries subscription after the timeseries. Example timeseries subscription configuration: ```yaml title="my_subscription.DatapointSubscription.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_subscription name: My Subscription description: All timeseries with externalId starting with ts_value partitionCount: 1 dataSetExternalId: ds_timeseries_hamburg filter: prefix: property: - externalId value: ts_value ``` ## Workflows *Requires Cognite Toolkit v0.2.0 or later* WorkflowTrigger requires Cognite Toolkit v0.3.0 or later **Resource directory: *workflows/*** **API documentation:** [Workflows](/api-reference/concepts/20230101/workflows) The workflows are stored in the module's `workflows/` directory. A workflow has three types of resources: `Workflow`, `WorkflowVersion`, and `WorkflowTrigger`. They are identified by the `Workflow.yaml`, `WorkflowVersion.yaml`, and `WorkflowTrigger` suffixes. We recommend having one file per workflow and workflow version. When creating, updating, and deleting workflows, the Cognite Toolkit applies changes in the correct order based on the dependencies between the workflows and workflow versions. The Cognite Toolkit creates transformations and functions before the workflow versions to ensure that the workflow versions can reference them. Example workflow: ```yaml title="my_workflow.Workflow.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: wf_my_workflow description: A workflow for processing data dataSetExternalId: ds_workflow_data ``` Example workflow version: ```yaml title="my_version.WorkflowVersion.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} workflowExternalId: wf_my_workflow version: '1' workflowDefinition: description: 'Run tasks in sequence' tasks: - externalId: '{{ workflow_external_id }}_function_task' type: 'function' parameters: function: externalId: 'fn_first_function' data: {} isAsyncComplete: false name: 'Task One' description: First task retries: 3 timeout: 3600 onFailure: 'abortWorkflow' - externalId: '{{ workflow_external_id }}_transformation_task' type: 'transformation' parameters: transformation: externalId: 'tr_first_transformation' concurrencyPolicy: fail name: 'Task Two' description: Second task retries: 3 timeout: 3600 onFailure: 'skipTask' dependsOn: - externalId: '{{ workflow_external_id }}_function_task' ``` Example workflow trigger: ```yaml title="my_trigger.WorkflowTrigger.yaml" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_trigger triggerRule: triggerType: schedule cronExpression: '0 0 * * *' input: my_input: 'data' metadata: origin: cognite-toolkit workflowExternalId: wf_my_workflow workflowVersion: '1' authentication: clientId: {{ my_trigger_clientId }} clientSecret: ${IDP_WF_TRIGGER_SECRET} ``` You can specify the credentials for the workflow trigger by adding a `authentication` property to the `WorkflowTrigger` configuration. ```yaml title="" wrap theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} externalId: my_trigger --- authentication: clientId: { { my_trigger_clientId } } clientSecret: ${IDP_WF_TRIGGER_SECRET} ``` The Cognite API doesn't support updating workflow Triggers. When you update a trigger with the Cognite Toolkit, the Cognite Toolkit deletes the existing trigger and creates a new one with the updated configuration. This means that the run history of the trigger is lost.