Building configurations

This article shows how to build local YAML files and check them before you deploy them to a Cognite Data Fusion (CDF) project. Data engineers use cdf build while they develop configuration. Cognite Toolkit administrators use it in CI/CD to stop a deploy when the build reports errors. Authenticate first and keep a .env file in the directory where you run the Cognite Toolkit. See Authenticating with the Cognite Toolkit. A small set of files can live directly in build/, which is what Deploying configurations does. Larger projects usually have more than one CDF project, small differences between those projects, and resources that depend on each other. cdf build organizes the YAML files, checks those dependencies, and substitutes variables so the same modules can target each project.

Create a module

A module is a folder of YAML files that you deploy together. One module can hold the configuration for ingesting data from one source. In this article, you create a small data model.Create modules/my_module/data_modeling/my.View.yaml with the following content:

my.View.yaml

space: my_space
externalId: MyView
version: v1
properties:
  name:
    container:
      type: container
      space: my_space
      externalId: MyContainer
    containerPropertyIdentifier: name

The directory looks like this:

modules/
  my_module/
    data_modeling/
      my.View.yaml

The directory name data_modeling and the suffix .View.yaml tell the Cognite Toolkit which resource type the YAML file defines. The resource library lists every supported resource.

Build the module

Run the following command to build the module:

cdf build

You did not specify which module to build, so the Cognite Toolkit prompts you to select one. Skip the prompt by naming the module: cdf build --modules my_module.

Select module to build

$ cdf build
? Which modules would you like to build? [my_module (modules/my_module)]

The Cognite Toolkit loads the module and builds it. In this example it loads one module and one resource file.

Review the build results

The Cognite Toolkit then validates agents, data modeling, dependencies, functions, and Cognite InField configuration.If the build finds insights, the Cognite Toolkit prints them:

Insights/issues

╭─ Build Insights ──────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                   │
│  ✗ Missing referenced resource  [MISSING-REFERENCED-RESOURCE]                                     │
│  ╰─ modules/my_module/data_modeling/my.View.yaml                                                  │
│     The space 'my_space' does not exist locally or in CDF. It is referenced by                    │
│     'my_space:MyView(version=v1)'.                                                                │
│     → Ensure that the space exists or remove the reference to it.                                 │
│                                                                                                   │
│  ✗ Missing referenced resource  [MISSING-REFERENCED-RESOURCE]                                     │
│  ╰─ modules/my_module/data_modeling/my.View.yaml:9:19                                             │
│     The container 'my_space:MyContainer' does not exist locally or in CDF. It is referenced by    │
│     'my_space:MyView(version=v1)'.                                                                │
│     → Ensure that the container exists or remove the reference to it.                             │
│                                                                                                   │
│  ✗ Missing referenced property  [MISSING-REFERENCED-PROPERTY]                                     │
│  ╰─ modules/my_module/data_modeling/my.View.yaml:10:34                                            │
│     Container property 'my_space:MyContainer.name' does not exist locally or in CDF. It is        │
│     referenced by 'my_space:MyView(version=v1)'.                                                  │
│     → Ensure that the container property exists or remove the reference to it.                    │
│                                                                                                   │
│  All insights are written to build/insights.csv                                                   │
│  To ignore a rule, add '# rules: ignore[CODE]' to the file, where [CODE] is the code found above  │
│  in brackets, or list the code under 'ignore' in the [rules] section of cdf.toml to ignore it in  │
│  all files                                                                                        │
│                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯

The build reports three insights for the view. You fix them in the Fix insights found in the build step. In most terminals, modules/my_module/data_modeling/my.View.yaml:10:34 is a link that opens the file at the line and column where the insight was found.The summary reports three errors and says Do not proceed to deploy. The view depends on a space and a container that are missing, so the deployment would fail.

Fix insights found in the build

Fix the insights from the Review the build results step by creating the missing space and container. Create modules/my_module/data_modeling/my.Space.yaml with the following content:

my.Space.yaml

space: my_space

and modules/my_module/data_modeling/my.Container.yaml with the following content:

my.Container.yaml

space: my_space
externalId: MyContainer
usedFor: node
properties:
  name:
    type:
      type: text
      list: false

The directory looks like this:

modules/
  my_module/
    data_modeling/
      my.Container.yaml
      my.Space.yaml
      my.View.yaml

Rebuild the module

Run the following command to rebuild the module:

cdf build --modules my_module

The build and validation match the Review the build results step. This time there are no insights, and the summary says Ready to deploy.Inspect the build/ directory. The Cognite Toolkit has created a subdirectory for each resource type and placed the built YAML files in the corresponding subdirectory. The built YAML files match the source YAML files.

Create config.dev.yaml

The space property in the space, container, and view is hardcoded to my_space in all three YAML files. That is easy to get wrong when the identifier changes, or when dev and prod use different spaces. Create a config.dev.yaml file:

config.dev.yaml

environment:
  name: dev
  project: my_cdf_project
  validation-type: dev
  selected:
    - modules/my_module

variables:
  space: my_space

For more information about the config.dev.yaml file, see Configuration file.The directory structure is now:

config.dev.yaml
modules/
  my_module/
    data_modeling/
      my.Container.yaml
      my.Space.yaml
      my.View.yaml

In the three YAML files, replace the hardcoded my_space with {{ space }}. For example, my.View.yaml:

my.View.yaml

space: {{ space }}
externalId: MyView
version: v1
properties:
  name:
    container:
      type: container
      space: {{ space }}
      externalId: MyContainer
    containerPropertyIdentifier: name

Rebuild the module with config.dev.yaml

Run the following command to rebuild the module with the config.dev.yaml file:

cdf build --config-yaml config.dev.yaml

The build and validation plan match the Rebuild the module step. In build/, the YAML files have space set to my_space.This command omits --modules. config.dev.yaml already lists the modules to build in environment.selected. If you pass --modules, that flag overrides environment.selected.

Next steps

Deploy the build/ directory. See Deploying configurations. For every field in config.[env].yaml, see Configuration file.

​Next steps

Next steps