Skip to main content

Setting up GitLab CI/CD pipelines with Cognite Toolkit

This article helps you set up CI/CD pipelines in GitLab to use the Cognite Toolkit to automate the deployment of your modules with version control.

Create a CI/CD pipeline​

To create a CI/CD pipeline:

  1. In the GitLab UI, open your repository. In the left menu, select Build > Pipeline editor.

  2. Click Configure pipeline to be guided through creating a .gitlab-ci.yml file to add to your repository to orchestrate the pipeline.

    The edit section shows a skeleton pipeline that you can configure. You can use this example CI/CD pipeline as a starting point:

    stages:
      - build
      - deploy_dry_run
      - deploy_to_cdf

    variables:
      CDF_CLUSTER: $CDF_CLUSTER
      CDF_URL: $CDF_URL
      CDF_PROJECT: $CDF_PROJECT
      CDF_CLIENT_TIMEOUT: 100
      IDP_CLIENT_ID: $IDP_CLIENT_ID
      IDP_CLIENT_SECRET: $IDP_CLIENT_SECRET
      IDP_TENANT_ID: $IDP_TENANT_ID
      CONTAINER_IMAGE: 'cognite/toolkit:0.5.35'

    build_modules:
      stage: build
      image: $CONTAINER_IMAGE
      script:
        - cdf build
      artifacts:
        paths:
          - build/
      rules:
        - if: '$CI_PIPELINE_SOURCE == "push"'
      retry: 2

    deploy_dry_run:
      stage: deploy_dry_run
      image: $CONTAINER_IMAGE
      script:
        - cdf deploy --dry-run
      dependencies:
        - build_modules
      rules:
        - if: '$CI_PIPELINE_SOURCE == "push"'
      retry: 2

    deploy_to_cdf:
      stage: deploy_to_cdf
      image: $CONTAINER_IMAGE
      script:
        - cdf deploy
      dependencies:
        - build_modules
      rules:
        - if: '$CI_COMMIT_BRANCH == "main"'
      retry: 2

Where:

  • stages defines the order of the jobs in a pipeline. A job has to finish successfully before the next one starts.

  • variables defines configuration data to authenticate to CDF. CONTAINER_IMAGE defines the version of the Cognite Toolkit to use for the jobs.

The next three blocks describe a job, in this example: build_modules, deploy_dry_run and deploy_to_cdf.

  • Use stage to define the name of the job.

  • Use script to pass the code to run during the job.

  • Within the build job, the artifacts section defines where build files are stored. Go to Build > Artifacts to see previous artifacts.

  • For deployment jobs (for example, deploy_dry_run), include a dependencies section. The deployment jobs depend on output from the previous job.

  • The rules section lists all rules for the job. In this example configuration, the rules are identical for build and deploy_dry_run, so these jobs run on push to any branch. deploy_to_cdf run only on pushes to main branch.

  • Use the retry section to define how many retry attempts to allow on failure. 2 is the maximum.

  1. Commit your .gitlab-ci.yml file to a branch. The jobs will fail until you finish the setup in the next section.

For more information, see the GitLab CI/CD pipelines documentation.

Create variables for running the pipeline​

To create variables:

  1. In the left-hand menu of your project, navigate to Settings > CI/CD.

  2. Select Variables.

  3. In empty table in the Project variables section, use the Add variable button to enter these variables with the correct values for your environment:

    • CDF_CLUSTER
    • CDF_PROJECT
    • CDF_URL
    • IDP_CLIENT_ID
    • IDP_CLIENT_SECRET (Visibility > "Masked and hidden")
    • IDP_TENANT_ID

  4. Important: Hide the IDP_CLIENT_SECRET value. Visibility for other variables can normally be set to "Visible."

Confirm that the pipeline runs as expected​

To check that the setup is successful, push a small change to a branch other than main. Follow the progress of your pipeline in Build > Pipelines.

Last updated: