Skip to main content
Jobs represent running hosted extractor processes that move data from sources to destinations. Each job links a source to a destination and applies a format to transform the data. You can create, monitor, update, and delete jobs. The API also provides access to job logs and metrics for monitoring extraction health.

​
What jobs are

A job connects a source (sourceId) to a destination (destinationId) and applies a format that controls how incoming data is converted before writing to CDF. The format is required and can be a built-in type (cognite, rockwell, value, sparkplug) or custom, which references a mapping by mappingId.
A mapping is required only when you use the custom format.
MQTT, Kafka, Event Hub, and MQTT broker jobs run continuously. REST jobs run on a fixed polling interval configured on the job.

​
Job lifecycle

  • Create: Specify a source, destination, format, and any source-specific config (such as a topic filter for MQTT or a path and interval for REST).
  • Monitor: Check the job’s status field, which reflects its current observed state (for example, running, connected, connection_error, transform_error). See Hosted extractors overview — Jobs for the full list of states and what each one means.
  • Update: Modify the source, destination, format, source-specific config, or set targetStatus to running or paused to start or pause the job.
  • Delete: Remove the job when extraction is no longer required.
Use job logs and metrics to detect failures, diagnose connectivity issues, and ensure data is flowing correctly.

​
Monitoring extraction

The API provides endpoints to retrieve job logs and metrics. Use these to identify errors and troubleshoot problems with sources or destinations.

​
Dead letter queue

The dead letter queue (DLQ) holds messages a job failed to transform. CDF stores them in CDF RAW. To use it, set dlqEnabled to true in the update jobs request. dlqEnabled is false by default. GET /hostedextractors/jobs/dlq lists the DLQ messages for a job. It requires hostedExtractors:READ and rawAcl:READ, because the request reads the messages with the caller’s credentials.
  • If the DLQ isn’t enabled for the job, the request fails with status 400 and the message DLQ is not enabled for job '<externalId>'.
  • If the job doesn’t exist, the request returns 404.
  • If RAW rejects the request, the 401 or 403 status passes through to the caller.
  • Errors from the underlying storage return 400 (for example, an invalid cursor), 429 (rate limited), 502 (storage unavailable), or 504 (storage timed out).
Last modified on November 2, 2026