Working with the Cognite Toolkit in a team

Use this page when several people change the Cognite Toolkit configuration for one Cognite Data Fusion (CDF) project. You will build several config files in one command, deploy only the modules Git shows as changed, and stop two people from deploying the same module at the same time.

These steps use alpha flags. The flags are experimental and may change or be removed without notice. Enable them on a shared development project. For staging and production, deploy with your continuous integration and continuous delivery (CI/CD) pipeline.

Before you begin, ensure you have:

Installed the Cognite Toolkit 0.9.9 or later for multiple config files, 0.9.21 or later for git diff selection, and 0.9.26 or later for deployment protection. Version details are on Alpha flags.
Completed Setting up the Cognite Toolkit locally.
Initialized a project repository with a cdf.toml file and environment configuration files.
Confirmed access to a Git repository and a CI/CD pipeline.

Understanding collaboration challenges

When you configure CDF projects across multiple sites, several team members often change configuration at the same time. Projects also span development, testing, staging, and production. Uncoordinated edits then overwrite each other’s work during merges and deployments, and resource definitions drift when people skip configuration best practices. This page shows how to prevent those conflicting changes. For inconsistent configurations, use the Cognite Toolkit validation rules. To promote a configuration from development to production, follow Setting up CI/CD for the Cognite Toolkit.

Setting up the team

To collaborate effectively when managing CDF configurations with the Cognite Toolkit, follow these recommended practices: Use version control. Manage all configuration files in a version control system such as Git. This tracks configuration history and allows you to revert changes when needed. Use a monorepo. Store all module and environment configurations in a single repository. A monorepo provides a single source of truth and simplifies dependency management between resources. Protect default branches. Enable branch protection on default branches (such as main). Require team members to propose changes through pull requests to ensure code review and automated validation before merging. Automate with CI/CD. Implement a CI/CD pipeline to validate and deploy configuration changes automatically. Run automated checks on every pull request, and use a deployment pipeline on the default branch to publish changes to the target CDF environment.

Editing modules

Follow this workflow when modifying configuration modules:

Create a feature branch

Create a branch from the default branch (main). Use a descriptive branch name that follows your team’s naming conventions.

Edit the module

Edit only the module files relevant to your change in your local branch.

Create a pull request

Open a pull request targeting the default branch. Include a clear description of the proposed configuration changes.

Run automated checks

Verify that all automated CI/CD checks pass, including cdf build validation.

Request a review

Request a review from your team or designated module owners.

Apply requested changes

If reviewers request changes, commit and push updates to your feature branch. The pull request updates automatically.

Merge the pull request

After receiving approvals and passing all checks, merge the pull request into the default branch.

Deploy changes

The CI/CD pipeline deploys the merged configuration to the target CDF environment.Confirm that the pipeline run succeeds and that the resources in the target CDF project match the merged module files.

Building multiple config files

Teams often maintain a single config.[env].yaml file per environment, such as config.dev.yaml. When multiple team members edit the variables section at the same time, merge conflicts can occur. You can split that configuration into site-specific files, such as config.site1-dev.yaml and config.site2-dev.yaml. Building those files in separate commands can produce conflicting resource definitions. Build them in one cdf build command so the Cognite Toolkit checks that the resources agree.

Enable multiple config files

Set the multi-configs alpha flag in cdf.toml:

cdf.toml

[alpha_flags]
multi-configs = true

Build the config files together

Pass each file to cdf build. Use the same environment.project and the same environment.validation-type in every file. environment.selected and variables stay independent, so each file can build the same modules with different variable values.

cdf build --config-yaml config.site1-dev.yaml --config-yaml config.site2-dev.yaml

Confirm the build

cdf build completes. The Cognite Toolkit validates the resources from every config file in that command. When environment.project or environment.validation-type differs between the files, align those fields and run the command again.

Selecting changed modules with git diff

Git diff selection builds and deploys only modules whose YAML files changed relative to deploy-branch. Enable it with the git-diff-select alpha flag, then set selection-type: git-diff. Use git diff selection on a shared development project. In staging and production, deploy every configured module through your CI/CD pipeline. Selecting every module on each deploy can overwrite work that other people still have on feature branches. This config.dev.yaml does that:

config.dev.yaml

environment:
  name: dev
  project: my-dev-project
  validation-type: dev
  selected:
    - modules/

With this configuration, every merge to the default branch builds and deploys all modules. Commenting modules out of environment.selected by hand is easy to get wrong:

config.dev.yaml

environment:
  name: dev
  project: my-dev-project
  validation-type: dev
  selected:
    # - modules/another_module
    - modules/my_module

Enable git diff selection

Set the git-diff-select alpha flag in cdf.toml:

cdf.toml

[alpha_flags]
git-diff-select = true

Point selection at the deploy branch

In the environment configuration file, set selection-type to git-diff, set validation-type to dev, and set deploy-branch to the branch you deploy from. The Cognite Toolkit rejects selection-type: git-diff when validation-type is anything other than dev.

config.dev.yaml

environment:
  name: dev
  project: my-dev-project
  validation-type: dev
  selection-type: git-diff
  deploy-branch: main

Commit the module YAML

Commit the changed .yaml and .yml files under modules/ before you build. Git diff selection reads committed files. When those files are still uncommitted, cdf build stops and lists them.

Build and deploy the changed modules

Run cdf build, then cdf deploy.On a feature branch, the Cognite Toolkit runs git diff against deploy-branch and builds only the modules that differ.On deploy-branch, the Cognite Toolkit runs git diff against the previous commit. Merge pull requests into that branch with Squash and merge. Otherwise, that diff cannot detect the full set of changed modules.

Confirm the selection

Run cdf build --verbose. The Selection section lists only the module paths that differ from deploy-branch.When no module YAML differs from deploy-branch, the command stops with No modules were found to build based on the git diff selection.

Protecting deployments from concurrent overwrites

Deployment protection stops two people from deploying the same module to a shared development project at the same time. Enable it with the deploy-protection alpha flag, then set deploy-protection: protected. You can use it with or without git diff selection. In staging and production, run deployments through your CI/CD pipeline. Deployment protection is for shared development projects. It leaves access control unchanged, and a person who already holds the lock can deploy again from that same branch.

Enable deployment protection

Set the deploy-protection alpha flag in cdf.toml:

cdf.toml

[alpha_flags]
deploy-protection = true

Protect the development environment

In the environment configuration file, set deploy-protection to protected and set deploy-branch.

config.dev.yaml

environment:
  name: dev
  project: my-dev-project
  validation-type: dev
  deploy-branch: main
  deploy-protection: protected

To combine deployment protection with git diff selection, add selection-type to that same environment block:

config.dev.yaml

environment:
  name: dev
  project: my-dev-project
  validation-type: dev
  selection-type: git-diff
  deploy-branch: main
  deploy-protection: protected

Deploy from your branch

Run cdf deploy from a Git repository.On a feature branch, the Cognite Toolkit requests a lock for each module in the deployment. It grants the lock when the module is unlocked, the lock is already on your current branch, or the lock is more than 24 hours old. When another branch holds a current lock, cdf deploy stops and deploys nothing.On deploy-branch, cdf deploy takes the locks for the modules in the deployment and releases them when the deployment finishes.

Confirm the locks

A successful deploy prints a panel titled Successfully acquired locks. followed by the module identifiers.When another branch holds a lock, the command stops with Failed to acquire locks for modules: and names the module, the Git user who holds the lock, and that user’s branch.After a successful deploy on deploy-branch, the Cognite Toolkit prints a panel titled Successfully released locks.

Configure and deploy

Working with the Cognite Toolkit in a team

Understanding collaboration challenges

Setting up the team

Editing modules

Building multiple config files

Selecting changed modules with git diff

Protecting deployments from concurrent overwrites

Further reading

​Understanding collaboration challenges

​Setting up the team

​Editing modules

​Building multiple config files

​Selecting changed modules with git diff

​Protecting deployments from concurrent overwrites

​Further reading

Understanding collaboration challenges

Setting up the team

Editing modules

Building multiple config files

Selecting changed modules with git diff

Protecting deployments from concurrent overwrites

Further reading