Before you start
- A data set must exist for the data you want to add to an extraction pipeline.
- Navigate to Access and set any of these capabilities for users, extractors, and third-party actors, such as GitHub Actions:
| User | Action | Capability | Description |
|---|---|---|---|
| End-user | Create and edit extraction pipelines | extractionpipelines:write | Gives access to create and edit individual pipelines and edit notification settings. Ensure that the pipeline has read access to the data set being used by the extraction pipeline. |
| View extraction pipelines | extractionpipelines:read | Gives access to list and view metadata of the pipeline. | |
| Create and edit extraction configurations | extractionconfigs:write | Gives access to create and edit an extractor configuration in an extraction pipeline. | |
| View extraction configurations | extractionconfigs:read | Gives access to view an extractor configuration in an extraction pipeline. | |
| View extraction logs | extractionruns:read | Gives access to view run history reported by the extraction pipeline runs. | |
| Extractor | Read extraction configurations | extractionconfigs:read | Gives access to read an extractor configuration from an extraction pipeline. |
| Post extraction logs | extractionruns:write | Gives access to post run history reported by the extraction pipeline runs. | |
| Third-party actors | Create and edit extraction pipelines | extractionpipelines:write | Gives access to create and edit individual pipelines and edit notification settings. Ensure that the pipeline has read access to the data set being used by the extraction pipeline. |
| Create and edit extraction configurations | extractionconfigs:write | Gives access to create and edit the extractor configuration from an extraction pipeline. |
Create extraction pipelines
1
Navigate to extraction pipelines
Navigate to Data management > Integrate > Extraction pipelines, orData management > Data catalog. Then select a data set and open the Lineage tab to add a pipeline to the selected data set.
2
Create the pipeline
Select Create extraction pipeline, where you will be requested to fill in the mandatory fields for creating a pipeline.
3
Open pipeline overview
Select Create to open the Extraction pipeline overview. On this page, you can add additional information to give contexts and insights about the pipeline.
You’ll see successful or failed runs when the connected extractor starts ingesting data into CDF. See the extractors’ configuration articles for setup.
Enable email notifications
Data owners and other stakeholders can receive email notifications about the extraction pipeline runs. The notifications are triggered when an extraction pipeline reports a failed run to CDF or an extraction pipeline with continuous data flow stops communicating with CDF. The notification is sent when a predefined time condition is reached.1
Add contact email
Under Contacts, enter the email address for the data owner.
2
Add additional contacts
Optionally, add other contacts for the extraction pipeline.
3
Enable notifications
Turn on the Notification toggle.
4
Confirm settings
Select Confirm.
Email notifications are only sent when an extraction pipeline status changes state or CDF has not registered any communication with the pipeline after a predefined time condition. This is to prevent multiple emails for ongoing incidents.For new incidents, emails are only sent for the first reported failed run and when the incident is resolved. Multiple reported failures in succession are ignored.
Edit the extractor configuration file
When you set up the Cognite extractors, you must create a configuration file that fits your requirements. Refer to the extractor documentation for details. You can create or edit the configuration in the Configuration file for extractor section to test and verify the settings, preferably in testing and staging environments. When applying the configuration to a production environment, we recommend setting up remote configuration files stored in the cloud using versioned files and continuous integration systems, such as GitHub Actions, or directly with the Cognite API.1
Create or edit configuration
Select Create configuration to create a file or copy and paste an existing file onto the canvas.
2
Publish configuration
Make your changes and select Publish to save. The extractor now reads the configuration from CDF.
3
Test settings
Test and verify the changed settings in the upcoming extractor run.
4
Deploy to production
Deploy the changed settings in a production environment, for instance, by committing the configuration file to GitHub for versioning and a continuous integration pipeline using GitHub Actions.