YAML reference library
The YAML resource configuration files are core to the Cognite Toolkit. Each of the files configures one of the resource types that are supported by the Cognite Toolkit and the CDF API. This article describes how to configure the different resource types.
The Cognite Toolkit bundles logically connected resource configuration files in modules, and each module stores the configuration files in directories corresponding to the resource types, called resource directories. The available resource directories are:
./<module_name>/
├── 3dmodels/
├── auth/
├── classic/
├── data_models/
├── data_sets/
├── extraction_pipelines/
├── files/
├── functions/
├── hosted_extractors/
├── locations/
├── raw/
├── robotics/
├── timeseries/
├── transformations/
└── workflows/
Note that a resource directory can host one or more configuration types. For example, the data_models/
directory hosts
the configuration files for spaces, containers, views, data models, and nodes. While the classic/
directory hosts the
configuration files for labels, assets, and sequences.
When you deploy, the Cognite Toolkit uses the CDF API to implement the YAML configurations in the CDF project.
In general, the format of the YAML files matches the API specification for the resource types. We recommend
that you use the externalId
of the resources as (part of) the name of the YAML file. This is to enable using the same resource
configuration across multiple CDF projects, for example, a development, staging and production project. Use number prefixes
(1.<filename.suffix>) to control the order of deployment within each resource type.
3D Models
Resource directory: 3dmodels/
Requires Cognite Toolkit v0.3.0 or later
API documentation: 3D models
3D model configurations are stored in the module's 3dmodels/ directory. You can have one or more 3D models in a single YAML file.
The filename must end with 3DModel
, for example, my_3d_model.3DModel.yaml
.
Example 3D model configuration:
name: my_3d_model
dataSetExternalId: ds_3d_models
metadata:
origin: cdf-toolkit
Assets
Resource directory: classic/
Requires Cognite Toolkit v0.3.0 or later
API documentation: Assets
Asset configurations are stored in the module's assets/ directory. You can have one or more asset in a single YAML file.
The filename must end with Asset
, for example, my_asset.Asset.yaml
.
Example Asset configuration:
externalId: my_root_asset
name: SAP hierarchy
description: The root asset in the SAP hierarchy
dataSetExternalId: ds_sap_assets
source: SAP
metadata:
origin: cdf-toolkit
The asset API uses an internal ID for the data set, while the YAML configuration files reference the external ID; dataSetExternalId
. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the CDF API.
Table formats
In addition, to yaml
Cognite Toolkit also supports csv
and parquet
formats for asset configurations. As with the yaml
format,
the filename must end with Asset
, for example, my_asset.Asset.csv
.
externalId,name,description,dataSetExternalId,source,metadata.origin
my_root_asset,SAP hierarchy,The root asset in the SAP hierarchy,ds_sap_assets,SAP,cdf-toolkit
Note that the column names must match the field names in the yaml
configuration. The exception is the metadata
field,
which is a dictionary in the yaml
configuration, but a string in the csv
configuration. This is solved by using the
notation metadata.origin
column in the csv
configuration.
Groups
Resource directory: auth/
API documentation: Groups
The group configuration files are stored in the module's auth/ directory. Each group has a separate YAML file.
The name
field is used as a unique identifier for the group. If you change the name of the group manually in CDF,
it will be treated as a different group and will be ignored by the Cognite Toolkit.
Example group configuration:
name: 'my_group'
sourceId: '{{mygroup_source_id}}'
metadata:
origin: 'cognite-toolkit'
capabilities:
- projectsAcl:
actions:
- LIST
- READ
scope:
all: {}
We recommend using the metadata:origin
property to indicate that the group is created by the Cognite Toolkit
Populate the sourceId
with the group ID from CDF project's identity provider.
You can specify each ACL capability in CDF as in the projectsAcl
example above. Scoping to dataset, space, RAW table,
current user, or pipeline is also supported (see ACL scoping).
Groups and group deletion
If you delete groups with the cdf clean
or cdf deploy --drop
command, the Cognite Toolkit skips the groups that the running
user or service account is a member of. This prevents the cleaning operation from removing access rights from
the running user and potentially locking the user out from further operation.
ACL scoping
Dataset scope
Use to restrict access to data in a specific data set.
- threedAcl:
actions:
- READ
scope:
datasetScope: { ids: ['my_dataset'] }
The groups API uses an internal ID for the data set, while the YAML configuration files reference the external ID. The Cognite Toolkit resolves the external ID to the internal ID before sending the request to the CDF API.
Space scope
Use to restrict access to data in a data model space.
- dataModelInstancesAcl:
actions:
- READ
scope:
spaceIdScope: { spaceIds: ['my_space'] }