Cognite Event Hub extractor
The features described in this section are currently in beta testing and are subject to change.
The Cognite Event Hub extractor connects to the Azure Event Hub and extracts data through an AMQP protocol.
The Event Hub extractor is hosted by Cognite, so you don't have to download and install anything. To deploy the extractor:
-
You need
hostedextractors:READ
andhostedextractors:WRITE
capabilities to create an Event Hub extractor. -
Make sure the extractor has the following access capabilities in a Cognite Data Fusion (CDF) project:
timeseries:READ
,timeseries:WRITE
for writing datapoints and time series,events:READ
,events:WRITE
,assets:READ
for writing events,raw:READ
,raw:WRITE
for writing raw rows, ordatamodelinstances:WRITE
for writing to data models.consejoYou can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. Read more.
-
Navigate to Data management > Integrate > Extractors.
-
Locate the Cognite Event Hub extractor and select Set up extractor.
-
You'll need a connection string to set up the Event Hub extractor:
- Locate your Event Hub in Azure.
- Navigate to Hub Settings > Built-in endpoints > Event Hub compatible endpoint.
About Azure Event Hub
Azure Event Hub is a data streaming service on Azure that supports payloads in any format. The Cognite Event Hub extractor supports a number of pre-defined message formats, or you can define your own.
Typically, it's sufficient to only create one single job per Event Hub source. An Event Hub is equivalent to a single topic on other streaming platforms. You can't filter data received from an Event Hub.
When you connect to an Event Hub, you specify a ConsumerGroup. You should create a new consumer group for each application that reads from your Event Hub.
Message formats
See Custom data formats for hosted extractors for how to write your own custom mapping of Event Hub messages.
Custom formats used with Event Hub jobs will receive an input
argument containing the message in a JSON format.