Configure, build, and deploy modules

voorzichtig

This section applies to Cognite Toolkit version 0.3.0. If you are using an earlier version, refer to the 0.2.0 section.

With the Cognite Toolkit, you first configure access to the CDF project(s) you'll administer. Next, you define and configure the modules you want to build and deploy. Then, you run the cdf-tk build command to build the necessary artifacts to deploy the modules.

Before you deploy the artifacts, we recommend that you always do a dry-run deployment to verify that the configurations and deployment perform as expected.

When you've verified the configuration, you run the cdf-tk deploy command to deploy the artifacts to the CDF project using the CDF SDK and APIs.

Step 1: Configure access to the CDF project

You must set up authentication for the Cognite Toolkit to access the target CDF project. Follow the steps below to configure the necessary access and create an environment file, .env, for the configuration.

Configuration prerequisites

Before you configure the Cognite Toolkit to access the CDF project, make sure these steps have been completed:

1. Register the CDF API and core application.

2. Set up groups to control access.

3. Register the Cognite Toolkit in your identity provider (IdP).

4. Add the Cognite Toolkit registration to an admin group.

   The Cognite Toolkit must be a member of a specific group that has a minimum of these capabilities in CDF.

   • projects:list and projects:read
   • groups:list, groups:read, groups:create, groups:update, and groups:delete

   If the group does not exists, the Toolkit will offer to create it for you.

Configure the Cognite Toolkit authentication

Follow these steps to configure the Cognite Toolkit and make sure that it has access to the CDF project:

1. Collect the values for the environment variables you need to configure:

   Description	Environment variable
   The cluster where your CDF project is running (for example, westeurope-1).	CDF_CLUSTER
   The CDF project name (e.g. myproject).	CDF_PROJECT
   The client ID of the application/service principal you created in your identity provider.	IDP_CLIENT_ID
   The client secret of the application/service principal you created in your identity provider.1	IDP_CLIENT_SECRET
   The token URL of your identity provider.2	IDP_TOKEN_URL

   ¹Set LOGIN_FLOW=interactive to use an interactive sign-in-flow. You'll be prompted to sign in through a browser instead of using a client secret.

   ²If you use Microsoft Entra ID as your IdP, you only need to specify the ID for your Entra ID tenant by replacing IDP_TOKEN_URL with IDP_TENANT_ID.

   info

   The Cognite Toolkit supports the CDF_TOKEN environment variable. If you have already created an OAuth2 token, for instance with Postman, you can use that token in the CDF_TOKEN environment variable, and only set the CDF_CLUSTER and CDF_PROJECT variables.

2. In a terminal, run this command to set up the authentication:

   cd <proj_dir>
   cdf auth init

   The Cognite Toolkit prompts you for the necessary information and asks you to store the information in a local .env file.

You can also use the .env.tmpl file to configure the environment variables manually, and then run the cdf auth verify to verify that you've set up the authentication correctly.

voorzichtig

The .env files can contain secrets — don't check them into version control repositories. The .env extension is added to the .gitignore file created in your project directory.

Step 2: Configure modules

To configure the modules to deploy to a CDF project:

1. The cdf modules init command will usually have created a config.<env>.yaml file for each of your environnments.

   opmerking

   You need one config.<env>.yaml file for each CDF project you deploy to. For example, if you have prod, staging, and dev projects/environments, you need config.prod.yaml, config.staging.yaml, and config.dev.yaml configuration files.

2. Open the configuration file, and edit the project property to match the name of your CDF project (line 3 in this example):

   environment:
     name: dev
     project: <customer-dev>
     type: dev
     selected:
       - cdf_demo_infield
       - cdf_oid_example_data

3. In the selected property, list the modules you want to deploy (lines 6-7 above).

   You can list any of the modules in the modules sub-directory.

4. Update the variables in the variables section to match the CDF project you're deploying to, for example, to change the name of your default location.

   important

   Make sure that you update all the variables that are set to <change_me>. If not, the module won't be deployed correctly.

5. Save the configuration file.

Step 3. Build deployment artifacts

To build the artifacts to deploy for the modules you have configured:

1. In a terminal, run these commands:

   cd <proj_dir>
   cdf build --env=dev

   The --env=<env> parameter specifies which configuration file to use, in this example config.dev.yaml.

The Cognite Toolkit substitutes the variables in the templates and creates a build/ output directory with the artifacts to deploy.

Step 4: Deploy to the CDF project

To deploy the configured artifacts to your CDF project:

1. In a terminal, run these commands to do a dry-run of the deployment:

   cdf deploy --dry-run

2. Inspect the output from the dry run and verify that the configurations and deployment perform as expected.

   

3. Run this command to deploy the artifacts to your CDF project:

     cdf deploy

To ensure that history and logs are not overwritten, the Cognite Toolkit deploys only configurations that have changed since the last run, and updates the configurations in the CDF project.

Clean up configurations and data

To deploy from a clean state, you can remove configurations and data before you deploy.

voorzichtig

Use these commands with caution to be sure that you do not accidentally delete configurations and data. Always run the commands first with the --dry-run option to inspect the potential impact.

Remove and redeploy configurations and data

To remove and redeploy configurations, but keep the data, run this command:

  cdf deploy --drop

To also remove and redeploy the data, add the --drop-data option:

  cdf deploy --drop --drop-data

To delete everything in your project that is managed by your configurations:

  cdf clean --dry-run