Message formats

When a hosted extractor receives a message from a source, it parses the payload using a message format you configure on the extraction job. The format determines how the extractor interprets the raw bytes and maps them to Cognite Data Fusion (CDF) resources such as time series data points. Message formats are independent of the source type. You can use any format with any source: MQTT, Kafka, Event Hub, or REST.

Built-in formats

Built-in formats handle common payload structures without requiring you to write a mapping expression. Each built-in format produces time series data points.

Format	Payload type	Use case
Simple value	Plain text	Single sensor readings where the topic or key identifies the time series
Cognite	JSON	Systems that produce the Cognite JSON standard for time series ingestion
Sparkplug B	Protobuf	Industrial IoT systems that use the Sparkplug B specification
Rockwell FactoryTalk	JSON	Rockwell FactoryTalk DataMosaix gateway data

All built-in formats share two optional configuration fields:

External ID prefix: A constant string prepended to every external ID the extractor generates.
External ID from topic: When enabled, the source topic or key is included in the external ID.

If both External ID prefix and External ID from topic are set, the generated ID is in the form [prefix][topic][id], where id is the field the format itself uses (for example, the metric name for Sparkplug B or the tag_id for Rockwell FactoryTalk). External ID from topic relies on the source providing a topic. MQTT and Kafka sources provide one; REST and Event Hub sources don’t, so External ID from topic has no effect for those sources.

Simple value

Use this format when each message is a single sensor reading as plain text, and the topic or key identifies which sensor produced the value. The extractor uses the topic as the external ID of the time series, the message content as the value, and the current time as the timestamp.

External ID construction

The topic is used as the external ID. External ID prefix is prepended when configured. External ID from topic has no additional effect for this format because the topic is already the ID.

Example message

32.5

Cognite

The Cognite message format is a JSON standard for transmitting time series data to CDF. Messages must be UTF-8 encoded JSON. The structure mirrors the CDF API for familiarity. Each message contains a timeseriesData array. Each entry specifies an externalId and an items array of data points with timestamp (millisecond Unix epoch) and value (number or string) fields. You can optionally include a status field on each data point to set a status code.

External ID construction

The externalId field of each timeseriesData entry is used as the external ID. When External ID from topic is enabled, the topic is prepended to it.

Example message

{
  "timeseriesData": [
    {
      "externalId": "my-timeseries",
      "items": [
        {
          "timestamp": 1676550645000,
          "value": 1.6
        },
        {
          "timestamp": 1676550663020,
          "value": 5.6
        }
      ]
    },
    {
      "externalId": "my-string-timeseries",
      "items": [
        {
          "timestamp": 1676550645000,
          "value": "on"
        },
        {
          "timestamp": 1676550663020,
          "value": "off",
          "status": "UncertainSubNormal"
        }
      ]
    }
  ]
}

Sparkplug B

Use this format for sources that publish Sparkplug B payloads. Sparkplug B is a binary protocol built on protobuf, widely used in industrial IoT for reporting device metrics over MQTT. The extractor decodes each metric in the protobuf payload and converts it to a CDF data point. The following metric value types are supported:

Sparkplug type	Protobuf type	CDF value type
Int	`uint32`	Numeric
Long	`uint64`	Numeric
Float	`float`	Numeric
Double	`double`	Numeric
Boolean	`bool`	String (`"true"` / `"false"`)
String	`string`	String (truncated to 255 characters)

Bytes, Dataset, Template, and Extension metric types are not supported and are skipped during ingestion.

Payload structure

Sparkplug B payloads are binary-encoded protobuf messages. Each payload contains a list of metrics, where each metric carries a name, a typed value, and an optional timestamp. The structure follows the Sparkplug specification:

message Payload {
    optional uint64  timestamp = 1;   // Payload-level timestamp (ms since epoch)
    repeated Metric  metrics   = 2;   // List of metrics
    optional uint64  seq       = 3;   // Sequence number
}

message Metric {
    optional string  name      = 1;   // Metric name (e.g., "Motor/Temperature")
    optional uint64  timestamp = 3;   // Metric-level timestamp (ms since epoch)

    oneof value {
        uint32  int_value      = 10;
        uint64  long_value     = 11;
        float   float_value    = 12;
        double  double_value   = 13;
        bool    boolean_value  = 14;
        string  string_value   = 15;
        // bytes, dataset, template, extension - not supported
    }
}

Timestamp resolution

The extractor uses the metric-level timestamp if present, then falls back to the payload-level timestamp, and finally to the current UTC time. Timestamps are in milliseconds since epoch.

External ID construction

The metric name is used as the external ID. When External ID from topic is enabled, the topic is prepended to the metric name.

Sparkplug reserves several metric names for protocol control (such as Node Control/Reboot, Node Control/Rebirth, and bdSeq). The extractor automatically filters out these metric names and does not ingest them as data points.

Rockwell FactoryTalk

Use this format for data from Rockwell FactoryTalk DataMosaix gateways. The extractor expects a JSON payload with a gatewayData array, where each entry contains a tag_id and an array of value-quality-timestamp (vqts) objects.

External ID construction

The tag_id field is used as the external ID. When External ID from topic is enabled, the topic is prepended to it.

Example message

{
  "gatewayData": [
    {
      "tag_id": "Plant1/Reactor/Temperature",
      "vqts": [
        {
          "v": 87.3,
          "t": "2024-01-15T10:30:00.000Z"
        },
        {
          "v": "88.1",
          "t": "2024-01-15T10:31:00.000Z"
        }
      ]
    }
  ]
}

Field mapping

Field	Description
`tag_id`	Used as the external ID of the time series.
`v`	The data point value. Accepts a JSON number or a string representation of a number.
`t`	Timestamp in RFC 3339 format (for example, `2024-01-15T10:30:00.000Z`).

The payload may include additional fields such as q (quality), model_id, and mimeType. These fields are ignored during ingestion. Each VQT entry becomes a separate data point in CDF.

Custom formats

When your payload structure doesn’t match a built-in format, define a custom format using mapping expressions. Custom formats let you parse the payload, extract fields, and map them to any supported CDF resource type. Supported resource types include data points, events, RAW rows, time series metadata, data model nodes and edges, and stream records. Custom formats also support additional input parsers beyond JSON: you can parse CSV, XML, and protobuf payloads before applying your mapping expression. For a complete guide to writing custom formats, see Custom message formats.

Preprocessing

Before the extractor applies a message format, it can run optional preprocessing steps on the raw payload bytes. These steps run in order: character encoding conversion first, then decompression.

Compression

If your source system sends compressed payloads, enable GZip decompression. The extractor decompresses the payload before passing it to the message format.

Character encoding

By default, the extractor expects UTF-8 encoded payloads. If your source system uses a different encoding, configure one of the following:

Encoding	Description
UTF-16 BE	UTF-16, big-endian byte order
UTF-16 LE	UTF-16, little-endian byte order
Latin-1	Windows CP-1252 (superset of ISO 8859-1)

The extractor converts the payload to UTF-8 before parsing.

Data engineering

Built-in formats

Simple value

External ID construction

Example message

Cognite

External ID construction

Example message

Sparkplug B

Payload structure

Timestamp resolution

External ID construction

Rockwell FactoryTalk

External ID construction

Example message

Field mapping

Custom formats

Preprocessing

Compression

Character encoding

Further reading

​Built-in formats

​Simple value

​External ID construction

​Example message

​Cognite

​External ID construction

​Example message

​Sparkplug B

​Payload structure

​Timestamp resolution

​External ID construction

​Rockwell FactoryTalk

​External ID construction

​Example message

​Field mapping

​Custom formats

​Preprocessing

​Compression

​Character encoding

​Further reading

Built-in formats

Simple value

External ID construction

Example message

Cognite

External ID construction

Example message

Sparkplug B

Payload structure

Timestamp resolution

External ID construction

Rockwell FactoryTalk

External ID construction

Example message

Field mapping

Custom formats

Preprocessing

Compression

Character encoding

Further reading