Cognite Event Hub extractor

Beta

The features described in this section are currently in beta testing and are subject to change.

The Cognite Event Hub extractor connects to the Azure Event Hub and extracts data through an AMQP protocol.

The Event Hub extractor is hosted by Cognite, so you don't have to download and install anything. To deploy the extractor:

1. You need hostedextractors:READ and hostedextractors:WRITE capabilities to create an Event Hub extractor.

2. Make sure the extractor has the following access capabilities in a Cognite Data Fusion (CDF) project: timeseries:READ, timeseries:WRITE for writing datapoints and time series, events:READ, events:WRITE, assets:READ for writing events, raw:READ, raw:WRITE for writing raw rows, or datamodelinstances:WRITE for writing to data models.

   conseil

   You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. Read more.

3. Navigate to Data management > Integrate > Extractors.

4. Locate the Cognite Event Hub extractor and select Set up extractor.

5. You'll need a connection string to set up the Event Hub extractor:

   1. Locate your Event Hub in Azure.
   2. Navigate to Hub Settings > Built-in endpoints > Event Hub compatible endpoint.

About Azure Event Hub

Azure Event Hub is a data streaming service on Azure that supports payloads in any format. The Cognite Event Hub extractor supports a number of pre-defined message formats, or you can define your own.

Typically, it's sufficient to only create one single job per Event Hub source. An Event Hub is equivalent to a single topic on other streaming platforms. You can't filter data received from an Event Hub.

When you connect to an Event Hub, you specify a ConsumerGroup. You should create a new consumer group for each application that reads from your Event Hub.

Message formats

See Custom data formats for hosted extractors for how to write your own custom mapping of Event Hub messages. Custom formats used with Event Hub jobs will receive an input argument containing the message in a JSON format.