Set up extraction pipelines with remote configuration files
You can set up the CDF extraction pipelines to use versioned extractor configuration files stored in the cloud. To deploy your extractors, you must supply a minimal configuration file containing the sign-in credentials and pull the remaining configuration from the cloud. You can generate the configuration files in the cloud with continuous integration systems such as GitHub Actions or directly with the CDF API.
Before you start
Make sure the extractor has the extractionconfigs:WRITE
capability to access CDF.
Configure extractors with GitHub Actions
-
Configure your extractor with a minimal configuration file. Refer to the extractor documentation for details. All the Cognite extractors have a similar configuration. This is an example for the Cognite DB extractor:
type: remote
cognite:
host: ${BASE_URL}
project: ${PROJECT}
idp-authentication:
client-id: ${CLIENT_ID}
secret: ${CLIENT_SECRET}
token-url: ${TOKEN_URL}
scopes:
- ${BASE_URL}/.default
extraction-pipeline:
external-id: db-extractor-pipeline