- Installed the Cognite Toolkit
0.9.9or later for multiple config files,0.9.21or later for git diff selection,0.9.26or later for deployment protection, and0.9.70or later for configuration file inclusion. Version details are on Alpha flags. - Completed Setting up the Cognite Toolkit locally.
- Initialized a project repository with a
cdf.tomlfile and environment configuration files. - Confirmed access to a Git repository and a CI/CD pipeline.
Understanding collaboration challenges
When you configure CDF projects across multiple sites, several team members often change configuration at the same time. Projects also span development, testing, staging, and production. Uncoordinated edits then overwrite each other’s work during merges and deployments, and resource definitions drift when people skip configuration best practices. This page shows how to prevent those conflicting changes. For inconsistent configurations, use the Cognite Toolkit validation rules. To promote a configuration from development to production, follow Setting up CI/CD for the Cognite Toolkit.Setting up the team
To collaborate effectively when managing CDF configurations with the Cognite Toolkit, follow these recommended practices: Use version control. Manage all configuration files in a version control system such as Git. This tracks configuration history and allows you to revert changes when needed. Use a monorepo. Store all module and environment configurations in a single repository. A monorepo provides a single source of truth and simplifies dependency management between resources. Protect default branches. Enable branch protection on default branches (such asmain). Require team members to propose changes through pull requests to ensure code review and automated validation before merging.
Automate with CI/CD. Implement a CI/CD pipeline to validate and deploy configuration changes automatically. Run automated checks on every pull request, and use a deployment pipeline on the default branch to publish changes to the target CDF environment.
Editing modules
Follow this workflow when modifying configuration modules:Create a feature branch
main). Use a descriptive branch name that follows your team’s naming conventions.Edit the module
Create a pull request
Run automated checks
cdf build validation.Request a review
Apply requested changes
Merge the pull request
Deploy changes
Building multiple config files
Teams often maintain a singleconfig.[env].yaml file per environment, such as config.dev.yaml. When multiple team members edit the variables section at the same time, merge conflicts can occur.
You can split that configuration into site-specific files, such as config.site1-dev.yaml and config.site2-dev.yaml. Building those files in separate commands can produce conflicting resource definitions. Build them in one cdf build command so the Cognite Toolkit checks that the resources agree.
Each file keeps its own environment.selected and variables. To merge files into one configuration, see Including configuration files.
Enable multiple config files
multi-configs alpha flag in cdf.toml:Build the config files together
cdf build. Use the same environment.project and the same environment.validation-type in every file. environment.selected and variables stay independent, so each file can build the same modules with different variable values.Confirm the build
cdf build completes. The Cognite Toolkit validates the resources from every config file in that command. When environment.project or environment.validation-type differs between the files, align those fields and run the command again.Including configuration files
Useinclude when several configuration files should become one configuration for a single cdf build. Team members can keep domain-specific files separate and share variables across those files.
To keep each file separate, with its own environment.selected and variables, build multiple config files in one command. With include, environment.selected is the union of every file, and the same variable key path in more than one file is an error.
When you enable config-include, the Cognite Toolkit merges the included files into the config.[env].yaml file that lists them:
environment.selectedis the union of the modules listed inselectedacross all files.- Any other
environmentfield must match the main file wherever an included file sets it. An included file can omitenvironmentwhen it only definesvariables. - The Cognite Toolkit merges
variablesmappings. Nested mappings combine, so each file can add its own keys. Setting the same key path in more than one file is an error. For example,modules.core.sourceIdin two files stops the build.modules.core.sourceIdin one file andmodules.site1.sourceIdin another file merge.
Enable configuration inclusion
config-include alpha flag in cdf.toml:Add include paths
include to your config.[env].yaml file. Each entry is a path to another configuration YAML file:.yaml or .yml file in that directory or a subdirectory. An included file can list further files in its own include field. The Cognite Toolkit loads each file once.Define the included configuration files
environment in a file that only sets variables:selected. Leave the other environment fields in the main file:environment field in an included file, it must equal the same field in the main file.Build the configuration
cdf build with your main environment file:environment.selected, and validates the resources. If the same variable key path is set in more than one file, or an environment field in an included file does not match the main file, the command stops with an error.Selecting changed modules with git diff
Git diff selection builds and deploys only the intersection of modules specified inenvironment.selected and modules whose YAML files changed relative to deploy-branch. Enable it with the git-diff-select alpha flag, then set selection-type: git-diff.
Use git diff selection on a shared development project. In staging and production, deploy every configured module through your CI/CD pipeline.
Selecting every module on each deploy can overwrite work that other people still have on feature branches. When environment.selected lists all modules, every deploy builds and publishes all of them:
environment.selected by hand is easy to get wrong:
environment.selected configured without manually commenting modules in and out. The Cognite Toolkit requires environment.selected to be set and only builds modules that are both listed in environment.selected and modified according to Git history.
Enable git diff selection
git-diff-select alpha flag in cdf.toml:Configure selection and the deploy branch
selection-type to git-diff, set validation-type to dev, set deploy-branch to the branch you deploy from, and configure selected. The Cognite Toolkit requires environment.selected to be set and rejects selection-type: git-diff when validation-type is anything other than dev.Commit the module YAML
.yaml and .yml files under modules/ before you build. Git diff selection reads committed files. When those files are still uncommitted, cdf build stops and lists them.Build and deploy the changed modules
cdf build, then cdf deploy.On a feature branch, the Cognite Toolkit runs git diff against deploy-branch and builds only the modules that are in environment.selected and differ from deploy-branch.On deploy-branch, the Cognite Toolkit runs git diff against the previous commit. Merge pull requests into that branch with Squash and merge. Otherwise, that diff cannot detect the full set of changed modules.Confirm the selection
cdf build --verbose. The Selection section lists only the module paths that are in environment.selected and differ from deploy-branch.When no changed module YAML matches environment.selected, the command stops with No modules were found to build based on the git diff selection.Protecting deployments from concurrent overwrites
Deployment protection stops two people from deploying the same module to a shared development project at the same time. Enable it with thedeploy-protection alpha flag, then set deploy-protection: protected. You can use it with or without git diff selection.
In staging and production, run deployments through your CI/CD pipeline. Deployment protection is for shared development projects. It leaves access control unchanged, and a person who already holds the lock can deploy again from that same branch.
Enable deployment protection
deploy-protection alpha flag in cdf.toml:Protect the development environment
deploy-protection to protected and set deploy-branch.selection-type and selected to that same environment block:Deploy from your branch
cdf deploy from a Git repository.On a feature branch, the Cognite Toolkit requests a lock for each module in the deployment. It grants the lock when the module is unlocked, the lock is already on your current branch, or the lock is more than 24 hours old. When another branch holds a current lock, cdf deploy stops and deploys nothing.On deploy-branch, cdf deploy takes the locks for the modules in the deployment and releases them when the deployment finishes.Confirm the locks
Successfully acquired locks. followed by the module identifiers.When another branch holds a lock, the command stops with Failed to acquire locks for modules: and names the module, the Git user who holds the lock, and that user’s branch.After a successful deploy on deploy-branch, the Cognite Toolkit prints a panel titled Successfully released locks.