Hosted extractors overview

A hosted extractor job reads from a source, transforms the data using a built-in format or a mapping, and writes to a destination. In order to create a hosted extractor, you need the hostedextractors:READ and hostedextractors:WRITE capabilities.

You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. Read more.

Sources

A hosted extractor source represents an external source system on the internet. The source resource in CDF contains all the information the extractor needs to connect to the external source system. A source can have many jobs, each streaming different data from the source system.

Jobs

A hosted extractor job represents the running extractor. Jobs produce logs and metrics that give the state of the job. A job can have nine different states:

State	Description
Paused	The job is temporarily stopped.
Waiting to start	The job is temporarily stopped and pending start. This state typically only lasts a few seconds.
Stopping	The job is running, but is supposed to stop. This state should only last for a few seconds at most.
Startup error	The job failed to start and will not attempt to restart. Check the configuration settings for the job to resolve this state.
Connection error	The job failed to connect to the source system and is currently retrying.
Connected	The job is connected to the source system, but has not yet received any data.
Transform error	The job is connected to the source system, received data but failed to transform and ingest the data into a CDF resource type.
Destination error	The job successfully transformed data, but failed to ingest data into a CDF resource type.
Running	The job is streaming data into CDF.

Job metrics

Jobs report metrics on their current execution. The following metrics are currently reported:

Metric	Description
Source messages	The number of input messages received from the source system.
Transform failures	The number of input messages that failed to transform.
Destination input values	The number of messages that successfully transformed and were given to destinations for uploading to CDF.
Destination requests	The number of requests made to CDF for this job.
Destination write failures	The number of requests to CDF that failed for this job.
Destination skipped values	The number of values that were invalid and were skipped before ingestion into CDF.
Destination failed values	The number of values that were not written to CDF due to failed requests.
Destination uploaded values	The number of values that were successfully ingested into CDF.

For the Cognite MQTT extractor, see Message delivery and reliability for details about message delivery during disconnections.

Email notifications

The feature described in this section is currently in public preview testing and is subject to change. Validate behavior in non-production environments first.

Hosted extractor jobs expose state and metrics in CDF. Email notifications tell operators when a job enters an operational state you care about (for example an error or paused state) or returns to a healthy state. Notifications are delivered through the Signals API. Receive email notifications when a hosted extractor job changes status by creating signal sinks (email delivery targets) and signal subscriptions that match job status signals. Users with the appropriate access capabilities can configure notifications for any user. Scope All lets you configure notifications broadly and Current user limits configuration to yourself.

The Signals API is documented as preview. Send the cdf-version header your environment expects and review Beta API reference before you depend on this in production.

Destinations

A hosted extractor writes to a destination. The destination only contains credentials for CDF. Multiple jobs can share a single destination, in which case they will make requests together, reducing the number of requests made to the Cognite APIs. Metrics will still be reported individually. For permanent deployments, the destination should be given a dedicated set of credentials. The credentials given to a destination should only grant access to the CDF resources the extractor needs to write based on the job’s mapping. See Data formats for details on mappings.

Supported output types

Depending on the job format or mapping, a job can write to these CDF resource types:

Datapoints and time series
Events
RAW table rows
Data model instances (nodes and edges)
Records in streams

Writing to Records is a public preview feature. Use a custom mapping with the immutable_record output type and set beta to true. For an end-to-end setup example, see Build a stream-to-stream transformation pipeline. This example covers source, destination, mapping, and job configuration for a hosted extractor that reads from one stream and writes transformed records to another.

Access capabilities

The CDF user creating an instance of a hosted extractor needs both the hostedextractors:READ and hostedextractors:WRITE capabilities.

Output type	Required capabilities
Datapoints	timeseries:READ and timeseries:WRITE. datamodelinstances:WRITE if you write to a data model extended from the `CogniteTimeSeries` data model type.
Timeseries	timeseries:READ and timeseries:WRITE. datamodelinstances:WRITE if you write to a data model extended from the `CogniteTimeSeries` data model type.
Events	events:READ and events:WRITE. assets:READ if you produce events linked to assets.
Raw	raw:WRITE
Data Models	datamodelinstances:WRITE
Records	streamrecords:WRITE and datamodels:READ scoped to each space that contains a source container.

For extra security, you can scope the access of the extractor to the data sets/spaces it writes to.

Granting hostedextractors:WRITE to a user may cause a privilege escalation. Any user that has access to create hosted extractors also has access to all the resources any configured destinations have access to.

Mapping

A mapping is a custom transformation, translating the source format to a format that can be ingested into CDF. Read more in Custom data formats for hosted extractors.

​Sources

​Jobs

​Job metrics

​Email notifications

​Destinations

​Supported output types

​Access capabilities

​Mapping

Sources

Jobs

Job metrics

Email notifications

Destinations

Supported output types

Access capabilities

Mapping