Working with modules - Cognite Docs

cdf modules lists the modules in your project, pulls CDF changes back into those modules, adds deployment packs, and upgrades project files when you install a new Cognite Toolkit version.

List modules

Use this command to see which modules are in the modules/ directory and whether the Cognite Toolkit found any issues. Data engineers and Cognite Toolkit administrators both run it. cdf modules list prints one row per module:

Module status

Module details
 Module     Status    Resources   Insights                   Syntax warnings
─────────────────────────────────────────────────────────────────────────────
 access     SUCCESS           1   ✓                                        0
 my_module  SUCCESS          23   ✓                                        0

SUCCESS and a check mark under Insights mean the module is ready to build and deploy.

Pull modules

Use cdf modules pull to copy resources from your CDF project back into modules/. The command keeps the module structure, variables, and comments. It only updates resources that are already in a module. It is the reverse of cdf build and cdf deploy. Pull after you edit a resource in the CDF UI and want that change in the module. Before you pull, read Building configurations and Deploying configurations. Keep a .env file for your CDF project.

Create a bare-bones transformation

Build and deploy a transformation to CDF. Create this YAML file:

Bare-bones transformation

externalId: {{transformationPrefix}}myTransformation
name: myTransformation
ignoreNullFields: true # Keep null fields from the source data.

and place it in modules/my_module/transformations/my.Transformation.yaml.

modules/
  my_module/
    transformations/
      my.Transformation.yaml

Build and deploy the transformation:

cdf build --config-yaml config.dev.yaml

and

cdf deploy

Edit the transformation in CDF

Open the transformation in CDF and add this query:

select * from cdf_nodes('cdf_cdm', 'CogniteDescribable', 'v1')

Pull the transformation into your local module

Pull the transformation from CDF into the local module:

cdf modules pull --config-yaml config.dev.yaml my_module

The command writes the CDF change into the local module and keeps the module structure, variables, and comments. The summary reports one changed transformation.

Inspect result

Open the transformation YAML:

Pulled transformation

externalId: {{transformationPrefix}}myTransformation
name: myTransformation
ignoreNullFields: true # Keep null fields from the source data.
query: select * from cdf_nodes('cdf_cdm', 'CogniteDescribable', 'v1')

The query you added in CDF is now in the local module. The variable, the comment, and the rest of the file are unchanged.If you use git or another version control system, you should now commit the changes to your local module.

Add deployment packs

Data engineers rarely write every resource from scratch. Start from a deployment pack: a templatized module you adapt for your project. See Deployment packs.

Set the deployment pack source

Put cdf.toml in the directory where you run cdf. The file includes the source of the deployment packs.

Example cdf.toml file

...

[library.cognite]
url = "https://github.com/cognitedata/library/releases/download/latest/packages.zip"

This is the default source of deployment packs. Cognite develops and maintains that library.

Add a deployment pack

To add a deployment pack, run:

cdf modules add

If you do not have a modules/ directory, run cdf modules init instead. cdf init runs that command when you select Modules.The wizard asks you to select and install a deployment pack:

Select a deployment pack

$ cdf modules add
Adding library cognite from https://github.com/cognitedata/library/releases/download/latest/packages.zip
Download ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Unzipping ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

? Which package would you like to use? Common: Core platform setup: common resources, ingestion pipeline, and search. (3)
   Which modules in common would you like to add? [Ingestion Workflow]

You have selected the following:

     modules
     └── common
         └── cdf_ingestion


? Would you like to make changes to the selection? No
  Creating common
    Creating module cdf_ingestion
  Updating config.dev.yaml
cdf.toml file already exists. Skipping creation.

Inspect the results

After the command has completed, you can inspect the results. Your modules/ directory should now contain the new module you added:

modules/
  common/
    cdf_ingestion/
      auth/
      data_sets/
      workflows/
      module.toml
      README.md

Each module has a module.toml file that describes the source of the module. Keep this file in the module directory. The README.md file contains documentation from the author of the deployment pack.Then inspect config.dev.yaml. If the deployment pack defines variables, config.dev.yaml is updated:

Example config.dev.yaml file

environment:
  ...

variables:
  ... Previous variables
  modules:
    common:
      cdf_ingestion:
        workflow: ingestion
        dataset: ingestion
        ...

The variables modules.common.cdf_ingestion.workflow and modules.common.cdf_ingestion.dataset are now in config.dev.yaml.

Next steps

The first step after adding a deployment pack is to update the variables to match your use case. Then build and deploy the resources in the module.

Upgrade modules

Cognite Toolkit administrators run cdf modules upgrade after installing a new CLI version. The command updates cdf.toml and CI/CD pipeline files so they match that version.

cdf modules upgrade --verbose

--verbose prints each change. A version change updates the Cognite Toolkit version in cdf.toml:

cdf.toml

[modules]
version = "0.9.24"

The same run updates the image tag in the workflow files from cognite/toolkit:0.9.23 to cognite/toolkit:0.9.24. Review the diff before you commit it.

When a resource format has a breaking change, cdf modules upgrade tries to rewrite the YAML files to the new format. Review that diff before you commit it.

​List modules

​Pull modules

​Add deployment packs

​Upgrade modules

List modules

Pull modules

Add deployment packs

Upgrade modules