> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka extractor

> Learn how to use the Cognite Kafka extractor to extract data from Apache Kafka brokers and topics. Cognite hosts the Kafka extractor, so you don't have to download or install anything.

## Before you start

* Assign [access capabilities](/cdf/access/guides/capabilities#cognite-kafka-extractor) to *create* a hosted Kafka extractor and for the extractor to *write* data points, time series, events, RAW rows, and instances in data models in the target CDF project.

<Tip>
  You can use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF data securely. [Read more](/cdf/integration/guides/extraction/admin_oidc).
</Tip>

## Deploy the extractor

<Steps>
  <Step title="Navigate to extractors">
    Navigate to <span class="ui-element">Data fusion</span> > <span class="ui-element">Integrate</span> > <span class="ui-element">Extractors</span>.
  </Step>

  <Step title="Set up the extractor">
    Locate the **Cognite Kafka extractor** and select **Set up extractor**.
  </Step>
</Steps>

Kafka is highly redundant, so when you connect to a Kafka broker, you typically connect to one or more from a list of *bootstrap brokers*. This tells the client which specific broker to connect to. You provide this list of bootstrap brokers to the extractor.

## Message formats

Kafka is an event streaming platform from Apache that supports payloads on any format. The Cognite Kafka extractor supports several [pre-defined message formats](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_mappings).

If you want to define your own custom mapping of Kafka messages, see [custom data formats for hosted extractors](/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). Custom formats used with Kafka jobs will receive an `input` argument containing the message as JSON, and a `context` argument containing the topic and some other message metadata, for example:

```
{
    "key": [1, 2, 3, 4],
	"topic": "stream_a",
    "headers": {
        "headerA": "valueA",
        "headerB": "valueB"
    },
    "timestamp": 1710312447698,
    "current_offset": 5912,
    "partition_id": 1
}
```

Messages are published on *topics*. A Kafka broker has a fixed set of topics, each writing data to one or more *partitions*. When you create a Kafka extractor instance, you must specify the *number* of partitions and the *name* of the topic.

This information can be used to help construct data points or other supported output types.