REST extractor - Cognite Docs

The REST extractor uses the Cognite mapping language to implement custom pagination, incremental load, and to handle responses. See custom mappings for details on writing your mappings.

Before you start

Assign access capabilities to create a hosted REST extractor and for the extractor to write data points, time series, events, RAW rows, and instances in data models in the target CDF project.

You can use OpenID Connect and your existing identity provider (IdP) framework to securely manage access to CDF data. For more information, see setup and administration for extractors.

Deploy the extractor

Open hosted extractors

Navigate to Data fusion > Integrate > Extraction pipelines > Hosted extractors.

Set up the extractor

Select Set up hosted extractor, choose Cognite REST extractor, then select Set up extractor.

When to use the hosted REST extractor

REST is a generic system for creating APIs over HTTP. The hosted Cognite REST extractor ingests data from a REST API on a fixed schedule with custom mappings and pagination. Use it when your API fits the following scope:

The extractor makes requests to a single host. The exception is that the host specified under authentication may be different.
The extractor sends request payloads as a valid JSON blob.
The extractor stores the last_run field as the singular state value between executions. This value indicates the most recent execution of the REST extractor in your project.

Use the Cognite Extractor Utils library and develop a custom REST extractor for your source system when you need to make requests to multiple hosts, combine source data or information in complex ways, or read data from CDF. Use the hosted REST extractor when you want to list information from a REST API and ingest it into CDF resources without building a custom extractor.

Configure the extractor

The extractor configuration consists of several parts required to extract from a REST source.

Create the source

Host is the hostname or IP address the REST extractor connects to. Enter only the hostname, not the full URL. For example, for https://api.cognite.com/v1/project/test/assets, the host is api.cognite.com.
Scheme is the type of connection to use, either http or https. Most public APIs use https.
Port is the port on the source server. Note that http and https typically use standard ports 80 and 443. Leave the port blank unless your source system indicates a different port.
CA certificate trusts the API server when it uses a self-signed or privately signed TLS certificate. Upload the public certificate of that certificate authority. This is not the same as an authentication certificate, which presents a client certificate to the API.
Authentication controls how the extractor authenticates to the API. Unlike manually setting headers, using the authentication option ensures that passwords and secrets are not visible when the source is read, and that they are stored with additional encryption. Choose one of the following:
- Basic authentication, header value, query parameter, or client credentials.
- Use an authentication certificate — present a client certificate to the API (mutual TLS). Upload an authentication certificate and an authentication certificate key in PEM or DER format. If the key is encrypted, enter the password for the certificate key.
- Use basic authentication and certificate — username and password plus a client certificate when the API requires both.

After you create a source, open it from Hosted extractors, go to Authentication details, and select Edit to update credentials or certificate files.To configure sources with the API, including authCertificate, see Sources.

Create the job

interval represents how often the job should run. You can pick any of the options from the fixed list of intervals.
path is the path to the API endpoint portion of the URL. It does not include the host information or the query parameters. For https://api/cognite.com/v1/projects/test/assets?externalId=test the path portion will be /v1/projects/test/assets.
method is the HTTP method for the request. The default is get. Sending a JSON body is only permitted if the method is post.
body is the JSON body to send with post requests. When this contains information, the Content-Type header will also be added to the request.
query is a list of key/value pairs for the initial query portion of the URL. For example, when using /v1/projects/test/assets?externalId=test&name=something, the query is { "externalId": "test", "name": "something" }.
headers is the list of valid header key/value pairs representing the initial headers sent with the request.
incremental load is used to build the initial request based on previous runs of the extractor. Incremental load describes the configuration details. Mapping for incremental load is applied to the extractor startup. Incremental load avoids reading the same data every time the extractor runs.
pagination is used during a single extractor run to paginate through data received from the source. Pagination describes the configuration details. The pagination mapping is applied once for each request after the first and is used to determine whether to run more queries.

Mapping context

Like other hosted extractors, all mappings involved with hosted extractors are passed a context object. This uses the following format:

{
    "last_run": 12345678, // Timestamps since 01/01/1970 when this extractor last ran.
    "response": {
        "headers": { // List of headers in the last response.
            "key": "value"
        },
        "status": 201 // Last response message, will always be 2xx.
    },
    "request": {
        "path": "/last/path", // Last request path.
        "host": "myhost.com", // Last request host.
        "headers": { // List of headers in last request.
            "key": "value"
        },
        "query": { // List of query parameters in last request.
            "key": "value"
        },
        "method": "get", // Last request method.
        "body": {} // Last request body
    }
}

response will be empty for incremental load mappings, since there is no prior response when those are run.

Incremental load

Incremental load is applied to extractor startup. It may apply to query parameters, headers, or the body. The incremental load mapping is given only context as described in the mapping context, without anything in the response section. If the incremental load is set to use body, the result of the mapping will set the next message body. If you want to only modify the body given in the extractor configuration, you can use a mapping on the following form:

{
    ...context.request.body,
    ...{
        "set-this-field": "to-value"
    }
}

If set to use queryParam or header, the result will overwrite only the configured query parameter or header value. Example, for the CDF Assets API: type: body

{
    "filter": {
        ...context.request.body.filter,
        "lastUpdatedTime": {
            "min": try_int(context.last_run, 0)
        }
    }
}

Pagination

Pagination is applied after each request and used to determine whether the extractor should make more requests to the source. The pagination mapping is given the context as described in the mapping context, and the response body in body, is converted to JSON. If a pagination request returns the same value as the previous request or null, the extractor will stop paginating. This usually means you need to make sure to return null from the mapping if a cursor or similar is null. If pagination is set to use body, the result of the mapping will set the next message body. If you want to just modify the body given in extractor configuration or in incremental load, you can create a mapping using the following format:

{
    ...context.request.body,
    ...{
        "set-this-field": "to-value"
    }
}

If set to use queryParam or header, the result will overwrite only the configured query parameter or header value. Example, for the CDF Assets API: type: body

if(body.nextCursor, {
    ...context.request.body,
    "cursor": body.nextCursor
}, null)

This will set cursor in the body if nextCursor is set in the response and return null to end pagination otherwise.

​Before you start

​Deploy the extractor

​When to use the hosted REST extractor

​Configure the extractor

​Mapping context

​Incremental load

​Pagination

Before you start

Deploy the extractor

When to use the hosted REST extractor

Configure the extractor

Mapping context

Incremental load

Pagination