> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognite.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Cognite Data Fusion (CDF) is an industrial DataOps platform. Prefer English docs unless the user asks for another locale. > Follow /_llms/ index links recursively until you reach page URLs ending in .md. Fetch those Markdown twins instead of HTML. > For REST APIs, default to calendar version 20230101 (stable). Use 20230101-beta or 20230101-alpha only if the user is on a preview API. Cite paths under /api-reference/concepts// and the Cdf-Version header. Prefer the Python SDK (cognite-sdk) unless the user is in JavaScript/TypeScript. > Start here by task: platform overview /cdf/index.md; data modeling /cdf/dm/index.md; data integration /cdf/integration/index.md; access /cdf/access/index.md; CDF Toolkit /cdf/deploy/cdf_toolkit/index.md; REST quickstart /dev/quickstart.md; Python SDK https://cognite-sdk-python.readthedocs-hosted.com/en/stable/; Atlas AI /cdf/atlas_ai/concepts/index.md; Flows apps /cdf/flows/index.md; MCP and IDE setup /dev/guides/ide_ai_integration.md. > For implementation work, also load /skill.md. # Rotate external data source keys > > **Required capabilities:** `transformationsExternalDataSourcesAcl:WRITE` Rotate the RSA key pair for external data sources. The current key pair is removed and a new one is generated with the given expiry. The rotation is atomic across the request: if any item fails, no keys are rotated. Only Snowflake data sources are supported. If any item refers to a data source of another format, the whole request is rejected with a 400 and nothing is rotated. ## OpenAPI ````yaml https://storage.googleapis.com/cognitedata-api-docs/dist/20230101-alpha.json post /transformations/externaldata/rotatekeys openapi: 3.1.0 info: title: Cognite API description: >- Reference documentation for the Cognite Data Fusion (CDF) API. See the [API introduction](/api-reference/concepts/20230101/api-description) and [API changelog](/api-reference/changelog). version: v1 contact: name: Cognite Support url: https://support.cognite.com email: support@cognite.com servers: - url: https://{cluster}.cognitedata.com/api/v1/projects/{project} description: The URL for the CDF cluster to connect to variables: cluster: enum: - api - az-tyo-gp-001 - az-eastus-1 - az-power-no-northeurope - westeurope-1 - asia-northeast1-1 - gc-dsm-gp-001 default: api description: The CDF cluster to connect to project: default: publicdata description: The CDF project name. security: - oidc-token: - https://{cluster}.cognitedata.com/.default - oauth2-client-credentials: - https://{cluster}.cognitedata.com/.default - oauth2-open-industrial-data: - https://api.cognitedata.com/.default - oauth2-auth-code: - https://{cluster}.cognitedata.com/.default tags: - name: Token description: > Access tokens issued by an IdP (Azure AD, Google, etc.) are used to access CDF resources. - name: Advanced joins description: > **Note** This functionality is in alpha. Callers need to provide the `cdf-version: alpha` header to their requests. ## Concepts ### Advanced joins An **advanced join** is an abstraction for matching data entities, and enable subject-matter experts to input their knowledge into the matching process. They work natively on top of Cognite Data Fusion [data modeling](#tag/Data-modeling) capabilities. In a first iteration (Alpha version), advanced joins support finding candidates, getting input on, and populating _direct relations_ in views. ### Matches After creating the main advanced join object, the caller can input truth values by creating **matches**, which are attached to the main advanced join object. A match indicates that two data model instances should be linked, by materializing the start and end of the direct relation between the two linked instances. ### Matchers The main feature of advanced joins is to run several matching processes, and pick the best results to write back to the original data model. The matching processes are described by **matcher** objects in the API. For now, we support only [Raw](#tag/Raw) matchers: the caller can write outputs from the matching process of their choice, like [Entity Matching](#tag/Entity-matching) with different parameters. ### Jobs The API provides several capabilities once advanced joins and matches are created, like - **measuring the proportion of mapped instances** in the advanced join's view, - **estimating the quality of a matcher** using the matches (truth values) as reference, - **suggesting data quality improvements** by predicting which unmatched instances are most likely to increase data quality if matched, - **running an advanced join**, which means to populate the original instances' direct relation property with the best combination of matcher results and matches. - name: Assets description: >- The assets resource type stores digital representations of objects or groups of objects from the physical world. Assets are organized in hierarchies. For example, a water pump asset can be a part of a subsystem asset on an oil platform asset. ## Rate and concurrency limits Both the rate of requests (denoted as request per second, or ‘**rps**’) and the number of concurrent (parallel) requests are governed by limits, for all CDF API endpoints. If a request exceeds one of the limits, it will be throttled with a `429: Too Many Requests` response. More on limit types and how to avoid being throttled is described [here](https://docs.cognite.com/dev/concepts/resource_throttling). Limits are defined at both the overall API service level, and on the API endpoints belonging to the service.\ Some types of requests consume more resources (compute, storage IO) than others, and where a service handles multiple concurrent requests with varying resource consumption. For example, ‘CRUD’ type requests (**C**reate, **R**etrieve, **R**equest ByIDs, **U**pdate and **D**elete) are far less resource intensive than ‘Analytical’ type requests (List, Search and Filter) and in addition, the most resource intensive Analytical endpoint of all, Aggregates, receives its own request budget within the overall Analytical request budget.\ The version 1.0 limits for the overall API service and its constituent endpoints are illustrated in the diagram below.\ These limits are subject to change, pending review of changing consumption patterns and resource availability over time: ### Translating RPS into data speed A single request may retrieve up to 1000 items. In the context of Assets, 1 item = 1 asset record\ Therefore, the maximum theoretical data speed at the top API service level is 200,000 items per second for all consumers, and 150,000 for a single identity or client in a project. ### Use of Partitions / Parallel Retrieval As a general guidance, Parallel Retrieval is a technique that should be used where due to query complexity, retrieval of data in a single request session turns out to be slow. By parallelizing such requests, data retrieval performance can be tuned to meet the client application needs. Parallel retrieval may also be used where retrieval of large sets of data is required, up to the capacity limits defined for a given API service. For example (using the Assets API request budget): * A single request may retrieve up to 1000 items * Up to 23 requests per second may be issued for an analytical query (per identity), such as when using /list or /filter API endpoints * This provides a theoretical maximum of 23,000 items read per second per identity * The query complexity may result in it taking longer than 1s to read or write 1000 items in a single request * Therefore, it is appropriate to specify the query to retrieve a lower number of items per request, and retrieve more items in parallel, up to the theoretical maximum performance of 23,000 items per second. **Important Note:** Parallel retrieval should be only used in situations where, due to query complexity, a single request flow provides data retrieval speeds that are significantly less than the theoretical maximum.\ Parallel retrieval does not act as a speed multiplier on optimally running queries. Regardless of the number of concurrent requests issued, the overall requests per second limit still applies.\ So for example, a single request returning data at approximately 18,000 items per second will only benefit from adding a second parallel request, the capacity of which goes somewhat wasted as only an additional 5,000 items per second will return before the request rate budget limit is reached. - name: Data Modeling description: > Use the Data Modeling Service to build industrial knowledge graphs. For more information, see [Data modeling](https://docs.cognite.com/cdf/dm). This page contains the specifications for the 5 core data modeling resources: - Spaces - Instances (Nodes & Edges) - Containers - Views - Data Models ### URI formats used for lineage - Space: `cdf://{cluster}/{project}/domain/{domain}/models/{space}` - Data Model: `cdf://{cluster}/{project}/domain/{domain}/models/{space}/datamodel/{externalId}/{version}` - View: `cdf://{cluster}/{project}/domain/{domain}/models/{space}/view/{externalId}/{version}` - Container: `cdf://{cluster}/{project}/domain/{domain}/models/{space}/container/{externalId}` - Node: `cdf://{cluster}/{project}/domain/{domain}/models/{space}/node/{externalId}` - Edge: `cdf://{cluster}/{project}/domain/{domain}/models/{space}/edge/{externalId}` x-uri-format-fragments: - resource: Space uri-fragment: models/{space} - resource: Data Model uri-fragment: models/{space}/datamodel/{externalId}/{version} - resource: View uri-fragment: models/{space}/view/{externalId}/{version} - resource: Container uri-fragment: models/{space}/container/{externalId} - resource: Node uri-fragment: models/{space}/node/{externalId} - resource: Edge uri-fragment: models/{space}/edge/{externalId} - name: Streams description: > Use the streams and records API to build high-volume extensions to industrial knowledge graphs that are built with [Data Modeling](https://docs.cognite.com/cdf/dm). The streams API lets you manage the streams that are used for storing records. With the API you can create, list, retrieve, and delete streams, and view stream settings and statistics. A stream defines the data lifecycle, and not schema, type or source. Multiple different sets of data which have nothing in common can be put into the same stream, provided that settings of this stream fit lifecycle and usage patterns (volume, rate, etc) of the data involved. Streams can be either mutable or immutable, which affects how records can be modified, storage capacity limits, and query performance characteristics. Mutable streams allow records to be updated and deleted by the user. They provide consistent low-latency access and optimal query performance. However, the total number of records that can be stored is limited. Mutable streams are well-suited for use cases requiring frequent data access and updates. Immutable streams do not allow records to be updated or deleted by the user, but support ingestion of very large amounts of data. Query performance may vary depending on the age of the data being accessed, as the system optimizes storage over time. To delete all data in a stream, the stream itself should be deleted. To protect against irretrievable erroneous deletion, streams are 'soft deleted' allowing them to be recovered for up to 6 weeks after the time of deletion. The template used to create the stream determines the actual recovery time. A single project has a limited number of soft deleted streams at any given time. To avoid hitting this limit, please avoid using any pattern of create and delete streams in a high frequency. Please note that we expect streams to be long lived. The exception are streams created with one of the `test` templates. Deleting a stream can take a long time. How long depends on the stream settings and the volume of data stored. While a stream is soft deleted, it is not possible to recreate a stream with the same identifier as the deleted stream. Once a stream is deleted, it does not count as one of the active streams, and more streams can be created to serve as active streams. If a stream is accidentally deleted, it is possible to recover the data by contacting [Cognite Support](https://cognite.zendesk.com/hc/en-us/requests/new). You must contact Cognite no less than 1 week prior to the expiration of the stream retention period to ensure we can recover the data. ## Available stream templates **Note: Stream Templates are in Beta** The current Stream Templates are in beta. This means: - New templates may be added based on customer needs - Existing templates may be modified or removed if necessary - Such modifications will **not affect existing streams** created from these templates Choose your template carefully for production use, as templates cannot be changed after stream creation. This section lists all currently available templates that can be used for creating streams. ### Immutable streams #### ImmutableTestStream This template should be used exclusively for experimentation. It is configured for high throughput and total data volume, but has a short data retention period. Low retention in a soft-deleted state means that such streams can be quickly discarded when no longer needed or recreated to remove the experimental data. **Note:** This template should never be used for production purposes. As this template allows significant load on the system, if we detect improper usage patterns, we can change setting of streams created from this template as a last resort. - Maximum total number of records - 50 M (50,000,000) - Maximum total data volume - 50 GB - Data retention - 7 days - Maximum ingestion throughput (per 10 minutes) - 1.5 GB - Maximum reading throughput (per 10 minutes) - 1.5 GB - Maximum records ingested (per 10 minutes) - 800,000 items - Maximum unique properties with data across all records - 1000 - Maximum range filter interval for the `lastUpdatedTime` property - 7 days - Stream soft-delete retention (before hard delete) - 1 day - Maximum active streams per project - 3 #### BasicArchive This template is intended for perpetual storage of data. However, overall data volume is limited, which needs to be taken into account when planning usage. - Maximum total number of records - 50 M (50,000,000) - Maximum total data volume - 50 GB - Data retention - Unlimited (data never gets deleted) - Maximum ingestion throughput (per 10 minutes) - 170 MB - Maximum reading throughput (per 10 minutes) - 1.7 GB - Maximum records ingested (per 10 minutes) - 170,000 items - Maximum unique properties with data (across all records) - 1000 - Maximum range filter interval for the `lastUpdatedTime` property - 365 days - Stream soft-delete retention (before hard delete) - 6 weeks - Maximum active streams per project - 2 ### Mutable streams #### BasicLiveData This template is intended for production usage and offers significant data volume and throughput. - Maximum total number of records - 5 M (5,000,000) - Maximum total data volume - 15 GB - Maximum ingestion throughput (per 10 minutes) - 170 MB - Maximum reading throughput (per 10 minutes) - 500 MB - Maximum records ingested (per 10 minutes) - 170,000 items - Maximum records updated or deleted (per 10 minutes) - 85,000 items - Maximum unique properties with data (across all records) - 1000 - Stream soft-delete retention (before hard delete) - 6 weeks - Maximum active streams per project - 1 ## Rate and concurrency limits Both the rate of requests (denoted as request per second (‘**rps**’)) and the number of concurrent (parallel) requests are governed by limits, for all CDF API endpoints. If a request exceeds one of the limits, it will be throttled with a `429: Too Many Requests` response. See [Resource throttling](https://docs.cognite.com/dev/concepts/resource_throttling) for limit types and how to avoid throttling. As streams are intended to be long-lived, users are not expected to interact with these endpoints frequently. The version limits for the streams endpoints are illustrated in the tables below. These limits are subject to change, pending review of changing consumption patterns and resource availability over time:
Create and Delete request budget
Overall Per ID
Requests per second 2 1
Concurrent requests 1 1
Retrieve and List request budget
Overall Per ID
Requests per second 5 3
Concurrent requests 3 2
- name: Records description: >- Records are mutable or immutable data objects (depending on the stream template) stored in a stream. Records are created by ingesting data into a stream. Records are shaped similarly to instances in the Data Modeling service, and their schema is defined by the containers, or the views organizing those containers, referenced as sources. Each record is associated with a space. Space-based access control applies to records. This section specifies the `records` resource and uses the following Data Modeling concepts: - [Spaces](https://docs.cognite.com/cdf/dm/dm_concepts/dm_spaces_instances#space) - [Containers](https://docs.cognite.com/cdf/dm/dm_concepts/dm_containers_views_datamodels#containers) - [Views](https://docs.cognite.com/cdf/dm/dm_concepts/dm_containers_views_datamodels#views) A ```view``` can be used as a source for records if ```usedFor``` is set to ```record``` and it targets the ```stream``` you are working with (via its ```streamId``` attribute). Such a view works as a lookup table over the record containers it maps: view properties map (and may rename) container properties, and a view may embed a filter which is applied (in addition to the request filter) on every read through the view. Record views are created and managed with the [Data Modeling views endpoints](https://docs.cognite.com/api/v1/#tag/Views), not through this API. **Note:** Every `record` has a required top-level `externalId` property, which can be used in some queries to retrieve or aggregate records (for filtering or sorting). For mutable records, `externalId` (together with `space`) uniquely identifies a record for write operations (create, upsert, delete). For immutable records, the records API does not enforce uniqueness of the `space` + `externalId` pair, so multiple records may share the same identifier. ## Rate and concurrency limits Both the rate of requests (denoted as requests per second, or ‘**rps**’) and the number of concurrent (parallel) requests are governed by limits for all CDF API endpoints. If a request exceeds a limit, it will be throttled with a `429: Too Many Requests` response. See [Resource throttling](https://docs.cognite.com/dev/concepts/resource_throttling) for limit types and how to avoid throttling. Limits apply to the API endpoints for this service. Some request types consume more resources (compute, storage I/O) than others. For example, ingest requests are less resource-intensive than analytical requests (Aggregate, Retrieve). Limits for query endpoints (Sync, Retrieve, Aggregate) are hierarchical: - All query endpoints share a common **Query request budget** (shown in the tables below) - The Retrieve endpoint has an **additional** dedicated budget that is checked first - The Aggregate endpoint has an **additional** dedicated budget that is checked first The Sync endpoint only checks the Query request budget. The Retrieve and Aggregate endpoints must pass both their dedicated budget check AND the Query request budget check. For example, with mutable streams you can make up to 40 rps total across all query endpoints (Query budget limit), but only up to 20 of those can be Retrieve requests (Retrieve budget limit) and only up to 15 can be Aggregate requests (Aggregate budget limit). That means you could send 20 Retrieve + 15 Aggregate + 5 Sync = 40 total RPS. Query performance and rate limits vary between mutable and immutable streams because of their different storage characteristics. Mutable streams provide consistent high-performance queries and higher rate limits (see "mutable streams" limits in the tables below). Immutable streams are optimized for ingesting very large amounts of data, which results in lower query performance and stricter rate limits than mutable streams (see "immutable streams" limits in the tables below). When designing data access patterns, use mutable streams for high-performance queries and higher rate limits, or immutable streams when the priority is high-volume ingestion and long-term storage. The amount of data returned in responses from query endpoints (Sync, Retrieve, Aggregate) is also limited. Prefer reading only the data you need: use an appropriate `filter` and limit which `sources` are retrieved, rather than retrieving large result sets that you will not use. The version limits for the records endpoints are shown in the tables below. These limits are subject to change, pending review of consumption patterns and resource availability over time:
Ingest request budget
Overall Per ID
Requests per second 40 30
Concurrent requests 20 15
Query request budget
Overall Per ID
Requests per second: mutable streams 40 30
Concurrent requests: mutable streams 30 22
Requests per second: immutable streams 10 7
Concurrent requests: immutable streams 10 7
Response MB per second 4 3

Additional dedicated budgets for Retrieve and Aggregate endpoints:

The Retrieve and Aggregate endpoints have dedicated budgets that are checked in addition to the Query request budget shown above. A request to these endpoints must pass both budget checks.

Retrieve request budget
Overall Per ID
Requests per second: mutable streams 20 15
Concurrent requests: mutable streams 20 15
Requests per second: immutable streams 10 7
Concurrent requests: immutable streams 10 7

Note: Retrieve endpoint requests are checked against both this budget and the Query request budget above.

Aggregate request budget
Overall Per ID
Requests per second: mutable streams 15 12
Concurrent requests: mutable streams 10 7
Requests per second: immutable streams 5 4
Concurrent requests: immutable streams 5 4

Note: Aggregate endpoint requests are checked against both this budget and the Query request budget above.

Summary:

- name: Events description: >- Events objects store complex information about multiple assets over a time period. Typical types of events that would be stored in this service might include Alarms, Process Data, and Logs.\ For the storage of low volume, manually generated, schedulable activities (such as maintenance schedules, work orders or other ‘appointment’ type activities, the Data Modeling service is now recommended. #### Important Note: Events and Time Series are somewhat closely related in that both are high volume types of data, capable of recording data in microsecond resolutions. However, Events is not recommended as a Time Series store, such as where the data flow is from a single instance of sensors (i.e. temperature, pressure, voltage), simulators or state machines (on, off, disconnected, etc).\ Time Series offers very low latency read and write performance, as well as specialised filters and aggregations that are tailored to the analysis of time series data. An event’s time period is defined by a start time and end time, both millisecond timestamps since the UNIX epoch. The timestamps can be in the future. In addition, events can have a text description as well as arbitrary metadata and properties.\ When storing event information in metadata, it should be considered that all data is stored as string format. #### Note: In Events API, timestamps are treated as strings when added to metadata fields and aren’t converted to the user’s local time zone. **Caveats:**\ Due to the eventually consistent nature of Asset IDs stored in Events, it should be noted that Asset ID references obtained from the Events API may occasionally be invalid (such as if an Asset ID is deleted, but the reference to that ID remains in the Event record for a time). Asset references obtained from an event - through asset ids - may be invalid, simply by the non-transactional nature of HTTP. They are maintained in an eventual consistent manner. ## Rate and concurrency limits Both the rate of requests (denoted as request per second, or ‘**rps**’) and the number of concurrent (parallel) requests are governed by limits, for all CDF API endpoints. If a request exceeds one of the limits, it will be throttled with a `429: Too Many Requests` response. More on limit types and how to avoid being throttled is described [here](https://docs.cognite.com/dev/concepts/resource_throttling). Limits are defined at both the overall API service level, and on the API endpoints belonging to the service.\ Some types of requests consume more resources (compute, storage IO) than others, and where a service handles multiple concurrent requests with varying resource consumption. For example, ‘CRUD’ type requests (**C**reate, **R**etrieve, **R**equest ByIDs, **U**pdate and **D**elete) are far less resource intensive than ‘Analytical’ type requests (List, Search and Filter) and in addition, the most resource intensive Analytical endpoint of all, Aggregates, receives its own request budget within the overall Analytical request budget.\ The version 1.0 limits for the overall API service and its constituent endpoints are illustrated in the diagram below.\ These limits are subject to change, pending review of changing consumption patterns and resource availability over time: ### Translating RPS into data speed A single request may retrieve up to 1000 items. In the context of Events, 1 item = 1 event record\ Therefore, the maximum theoretical data speed at the top API service level is 200,000 items per second for all consumers, and 150,000 for a single identity or client in a project. ### Use of Partitions / Parallel Retrieval As a general guidance, Parallel Retrieval is a technique that should be used where due to query complexity, retrieval of data in a single request session turns out to be slow. By parallelizing such requests, data retrieval performance can be tuned to meet the client application needs. Parallel retrieval may also be used where retrieval of large sets of data is required, up to the capacity limits defined for a given API service. For example (using the Events API request budget): * A single request may retrieve up to 1000 items * Up to 23 requests per second may be issued for an analytical query (per identity), such as when using /list or /filter API endpoints * This provides a theoretical maximum of 23,000 items read per second per identity * The query complexity may result in it taking longer than 1s to read or write 1000 items in a single request * Therefore, it is appropriate to specify the query to retrieve a lower number of items per request, and retrieve more items in parallel, up to the theoretical maximum performance of 23,000 items per second. **Important Note:** Parallel retrieval should be only used in situations where, due to query complexity, a single request flow provides data retrieval speeds that are significantly less than the theoretical maximum.\ Parallel retrieval does not act as a speed multiplier on optimally running queries. Regardless of the number of concurrent requests issued, the overall requests per second limit still applies.\ So for example, a single request returning data at approximately 18,000 items per second will only benefit from adding a second parallel request, the capacity of which goes somewhat wasted as only an additional 5,000 items per second will return before the request rate budget limit is reached. - name: Files description: >- A file stores a sequence of bytes connected to one or more assets. For example, a file can contain a piping and instrumentation diagram (P&IDs) showing how multiple assets are connected. Each file is identified by the 'id' field, which is generated internally for each new file. Each file's 'id' field is unique within a project. The 'externalId' field is optional, but can also be used to identify a file. The 'externalId' (if used) must be unique within a project. Files are created in two steps; First the metadata is stored in a file object, and then the file contents are uploaded. This means that files can exist in a non-uploaded state. The upload state is reflected in the 'uploaded' field in responses. Asset references obtained from a file - through asset ids - may be invalid, simply by the non-transactional nature of HTTP. They are maintained in an eventual consistent manner. ## Rate and concurrency limits Both the rate of requests (denoted as request per second, or ‘**rps**’) and the number of concurrent (parallel) requests are governed by limits, for all CDF API endpoints. If a request exceeds one of the limits, it will be throttled with a `429: Too Many Requests` response. More on limit types and how to avoid being throttled is described [here](https://docs.cognite.com/dev/concepts/resource_throttling). Limits are defined at both the overall API service level, and on the API endpoints belonging to the service. Some types of requests consume more resources (compute, storage IO) than others, and where a service handles multiple concurrent requests with varying resource consumption. For example, ‘CRUD’ type requests (**C**reate, **R**etrieve, **R**equest ByIDs, **U**pdate and **D**elete and similar) are far less resource intensive than ‘Analytical’ type requests (List, Search and Filter) and in addition, the most resource intensive Analytical endpoint of all, Aggregates, receives its own request budget within the overall Analytical request budget. The version 1.0 limits for the overall API service and its constituent endpoints are illustrated in the diagram below. These limits are subject to change, pending review of changing consumption patterns and resource availability over time: ### Translating RPS into data speed A single request may retrieve up to 1000 items. In the context of Files, 1 item = 1 file record Therefore, the maximum theoretical data speed at the top API service level is 160,000 items per second for all consumers, and 120,000 for a single identity or client in a project. ### Use of Partitions / Parallel Retrieval As a general guidance, Parallel Retrieval is a technique that should be used where due to query complexity, retrieval of data in a single request session turns out to be slow. By parallelizing such requests, data retrieval performance can be tuned to meet the client application needs. Parallel retrieval may also be used where retrieval of large sets of data is required, up to the capacity limits defined for a given API service. For example (using the Files API request budget): * A single request may retrieve up to 1000 items * Up to 23 requests per second may be issued for an analytical query (per identity), such as when using /list or /filter API endpoints * This provides a theoretical maximum of 23,000 items read per second per identity * The query complexity may result in it taking longer than 1s to read or write 1000 items in a single request * Therefore, it is appropriate to specify the query to retrieve a lower number of items per request, and retrieve more items in parallel, up to the theoretical maximum performance of 23,000 items per second. **Important Note:** Parallel retrieval should be only used in situations where, due to query complexity, a single request flow provides data retrieval speeds that are significantly less than the theoretical maximum. Parallel retrieval does not act as a speed multiplier on optimally running queries. Regardless of the number of concurrent requests issued, the overall requests per second limit still applies. So for example, a single request returning data at approximately 18,000 items per second will only benefit from adding a second parallel request, the capacity of which goes somewhat wasted as only an additional 5,000 items per second will return before the request rate budget limit is reached. - name: Functions description: >- Functions enables Python code to be hosted and executed in the cloud, on demand or by using a schedule. Execution, status and logs are available through the API. A function is uploaded to the Files API as a zip file with at least a Python file called `handler.py` (unless specified otherwise through the `functionPath`-argument) that must contain a function named `handle` with any of the following arguments: `data`, `client`, `secrets`, or 'function_call_info', which are passed into the function. The latest version of Cognite SDK's are available, and additional python packages and version specifications can be defined in a `requirements.txt` in the root of the zip. - name: Function calls description: >- Function calls let you execute functions asynchronously with a timeout of 15 minutes. - name: Function schedules description: >- Function schedules allow you to run functions with a specific input at intervals defined by a cron expression. These function calls will be asynchronous and show up in the function call list. Visit http://www.cronmaker.com to generate a cron expression with a UI. - name: Function Apps description: | > **Private Beta:** This API is currently in private beta and available to > select customers. To request access, please reach out to your Cognite > contact or [Cognite Support](https://support.cognite.com/). Core operations for listing and retrieving Function Apps. - name: Function App Calls description: > > **Private Beta:** This API is currently in private beta and available to > select customers. To request access, please reach out to your Cognite > contact or [Cognite Support](https://support.cognite.com/). Invoke Function App endpoints and view call history, responses, and logs. All function invocations go through `POST /calls` with a `FunctionAppCallRequest` envelope containing the target path, HTTP method, body, and session nonce. - name: Function App MCP description: > > **Private Beta:** This API is currently in private beta and available to > select customers. To request access, please reach out to your Cognite > contact or [Cognite Support](https://support.cognite.com/). [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) endpoints for AI agent integration. Implements the [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports#streamable-http) with [JSON-RPC 2.0](https://www.jsonrpc.org/specification) messaging and SSE streaming. Nonce is passed per-call inside `params._meta["com.cognite/nonce"]` for methods that invoke the function (tools/list, tools/call, ping). - name: 3D description: >- The CDF 3D API organizes 3D data into models and revisions. A model is just a container for a set of revisions. Revisions contains the actual 3D data. For example you can have a model named `Compressor` and you can upload a revision under that model. When you create a revision you need to attach a file, which may be in 3D CAD or Point cloud format. For every new version of the 3D model you upload a new revision under the model. You can then easily track the history of a model by browsing the different revisions. When you upload a new revision the 3D API needs to process the 3D data to optimize it for rendering. This can take some time. The Get3D api get revision endpoint provides a status string in the revision object. You can then follow the process while you wait. A 3D CAD model is typically built up by a hierarchical structure. This is related to the layout of the asset hierarchy in the Cognite Asset API. The 3D API provides endpoints to extract the nodes from the 3D node hierarchy, and endpoints to make mappings from the 3D nodeIds to assetIds in the asset hierarchy. The nodes are assigned an ID, nodeId, based on the node name and hierarchy. Each node in the 3D hierarchy gets a unique ID. This is the ID representing the object in the viewer. When a user click on a object in the 3D viewer it can returns the ID for the object that was clicked. You can then use that ID to look up which node in the hierarchy the user clicked on. Cognite also provides a [web based 3D viewer](https://www.npmjs.com/package/@cognite/reveal) to embed the processed 3D model in your own web page app. - name: Time series description: >- A time series consists of a sequence of data points connected to a single asset. For example, a water pump asset can have a temperature time series that records a data point in units of °C every second. A single asset can have several time series. The water pump could have additional time series measuring pressure within the pump, rpm, flow volume, power consumption, and more.Time series store data points as either numbers or strings. This is controlled by the is_string flag on the time series object. Numerical data points can be aggregated before they are returned from a query (e.g., to find the average temperature for a day). String data points, on the other hand, can't be aggregated by CDF but can store arbitrary information like states (e.g., “open”/”closed”) or more complex information (JSON). Cognite stores discrete data points, but the underlying process measured by the data points can vary continuously. When interpolating between data points, we can either assume that each value stays the same until the next measurement or linearly changes between the two measurements. The `isStep` flag controls this on the time series object. For example, if we estimate the average over a time containing two data points, the average will either be close to the first (`isStep`) or close to the mean of the two (not `isStep`). A data point stores a single piece of information, a number or a string, associated with a specific time. Data points are identified by their timestamps, measured in milliseconds since the unix epoch -- 00:00:00.000, January 1st, 1970. The time series service accepts timestamps in the range from 00:00:00.000, January 1st, 1900 through 23:59:59.999, December 31st, 2099 (in other words, every millisecond in the two centuries from 1900 to but not including 2100). Negative timestamps are used to define dates before 1970. Milliseconds is the finest time resolution supported by CDF, i.e., fractional milliseconds are not supported. Leap seconds are not counted. Numerical data points can be aggregated before they are retrieved from CDF. This allows for faster queries by reducing the amount of data transferred. You can aggregate data points by specifying one or more aggregates (e.g., average, minimum, maximum) as well as the time granularity over which the aggregates should be applied (e.g., “1h” for one hour). Aggregates are aligned to the start time modulo the granularity unit. For example, if you ask for daily average temperatures since Monday afternoon last week, the first aggregated data point will contain averages for Monday, the second for Tuesday, etc. Determining aggregate alignment without considering data point timestamps allows CDF to pre-calculate aggregates (e.g., to quickly return daily average temperatures for a year). Consequently, aggregating over 60 minutes can return a different result than aggregating over 1 hour because the two queries will be aligned differently. Asset references obtained from a time series - through its asset ID - may be invalid simply by the non-transactional nature of HTTP. They are maintained in an eventually consistent manner. - name: Data point subscriptions description: >- A data point subscription is a way to listen to changes to time series data points, in ingestion order. A single subscription can listen to many time series, and a time series can be part of many subscriptions. You listen to subscriptions by calling the “list subscription data“ endpoint. It will return a list of data point upserts and deleted ranges, together with a cursor to retrieve the next batch of updates. Updates written since the creation of the subscription, up to 7 days ago, can be retrieved through the subscription. Subscriptions can be defined explicitly through a list of time series external ids and instance ids, or implicitly through a filter. The filter is a subset of the advanced filter query in the regular “time series search“ endpoint. Partitions: Subscriptions can be further divided into a number of partitions, for the purpose of fetching subscription data in parallel. Each partition will return data in ingestion order, but the order between partitions is not guaranteed. Limitations: - Each subscription can have at most 10000 time series. - A time series can be part of at most 10 subscriptions. - A project can have at most 1000 subscriptions, of which 100 are filter subscriptions. - Time series with security categories can not be shown in filter subscriptions. - Time series with neither external id nor instance id cannot be added to non-filter subscriptions. - The number of partitions cannot be changed after creation. Access control: You need READ access to subscriptions ACL to read/list subscriptions or list data, and you need WRITE access to create/update/delete subscriptions. Furthermore, you need READ access to the time series you want to get updates from. For filter subscriptions, you either need READ access to all time series, or the filter must be restricted according to your access rights (e.g. by filtering on specific data set IDs). Subscriptions can have data sets that allow for more granular access control. The data set on a subscription does not influence the data stream, but allows you to restrict who can read/update the subscription. Name of ACL: `timeSeriesSubscriptionsAcl` Supported actions: - `READ` - `WRITE` Example capabilities: ``` [ {"timeSeriesAcl":{"actions":["READ"], "scope":{"all":{}}}}, {"timeSeriesSubscriptionsAcl":{"actions":["WRITE", "READ"], "scope":{"all":{}}}} ] ``` - name: Synthetic Time Series description: >- Synthetic Time Series (STS) is a way to combine various input time series, constants and operators, to create completely new time series. For example can we use the expression `24 * TS{externalId='production/hour'}` to convert from hourly to daily production rates. But STS is not limited to simple conversions. * We support combination of different time series `TS{id=123} + TS{externalId='hei'} / TS{space='data modeling space', externalId='dm id'}`. * Functions of time series `sin(pow(TS{id=123}, 2))`. * Aggregations of time series `TS{id=123, aggregate='average', granularity='1h'}+TS{id=456}` * Convert time series with `unitExternalId` to another unit `TS{externalId='temp_c', targetUnit='temperature:deg_f'}`. To learn more about synthetic time series please follow [our guide](https://docs.cognite.com/dev/concepts/resource_types/synthetic_timeseries). - name: Raw description: > Manage data in the raw NoSQL database. Each project will have a variable number of raw databases, each of which will have a variable number of tables, each of which will have a variable number of key-value objects. Only queries on key are supported through this API.\ ### Request and concurrency limits Both the rate of requests and the number of concurrent (parallel) requests are governed by limits, for all CDF API endpoints. If a request exceeds one of the limits, it will be throttled with a `429: Too Many Requests` response. More on limit types and how to avoid being throttled is described [here](https://docs.cognite.com/dev/concepts/resource_throttling). The limits for the Raw service are described in the table below. Note that under high load, some deviation from the limits might occur for short periods of time as the service is scaling up. The `/rows` endpoints for inserting and retrieving data are governed by specific data rate limits. | Limit | Per project | Per user (identity) | |-----------------------|-----------------------|---------------------| | Concurrency | 64 parallel requests | 48 parallel requests| | Data rate (retrieve) | 8.3 GB / 10 minutes | 6.6 GB / 10 minutes | | Data rate (insert) | 1.6 GB / 10 minutes | 1.3 GB / 10 minutes | ### URI formats used for lineage - Database: `cdf://{cluster}/{project}/domain/{domain}/raw/{dbName}` - Table: `cdf://{cluster}/{project}/domain/{domain}/raw/{dbName}/{tableName}` x-uri-format-fragments: - resource: Database uri-fragment: raw/{dbName} - resource: Table uri-fragment: raw/{dbName}/{tableName} - name: Groups description: > Groups are used to give principals the capabilities to access CDF resources. One principal can be a member in multiple groups and one group can have multiple members. Note that having more than 20 groups per principal is not supported and may result in login issues. Groups can either be managed through the external identity provider for the project or managed by CDF. - **Group Membership Managed Externally**: Groups membership is managed by the external identity provider. It is not possible edit or see the members of these groups in CDF. - **Group Membership Managed within CDF**: Lets you see and edit group membership in CDF instead of relying on the external identity provider. - name: Projects description: > Projects are used to isolate data in CDF from each other. All objects in CDF belong to a single project, and objects in different projects are generally isolated from each other. - name: Security categories description: > Manage security categories for a specific project. Security categories can be used to restrict access to a resource. Applying a security category to a resource means that only principals (users or service accounts) that also have this security category can access the resource. To learn more about security categories please read [this page](https://docs.cognite.com/api-reference/concepts/20230101/security-categories). - name: Data sets description: >- Data sets let you document and track data lineage, ensure data integrity, and allow 3rd parties to write their insights securely back to a Cognite Data Fusion (CDF) project. Data sets group and track data by its source. For example, a data set can contain all work orders originating from SAP. Typically, an organization will have one data set for each of its data ingestion pipelines in CDF. A data set consists of metadata about the data set, and the data objects that belong to the data set. Data objects, for example events, files, and time series, are added to a data set through the `dataSetId` field of the data object. Each data object can belong to only one data set. To learn more about data sets, see [getting started guide](https://docs.cognite.com/cdf/data_governance/concepts/datasets/) - name: Limits description: Limits Service Endpoints to provide limit values for a project. - name: Metering description: Metering Service Endpoints to provide consumption data for a project. - name: Sequences description: > A sequence stores a table with up to 400 columns indexed by row number. There can be at most 400 numeric columns and 200 string columns. Each of the columns has a pre-defined type: a string, integer, or floating point number. For example, a sequence can represent a curve, either with the dependent variable x as the row number and a single value column y, or can simply store (x,y) pairs in the rows directly. Other potential applications include data logs in which the index isn't time-based. To learn more about sequences, see the [concept guide](https://docs.cognite.com/dev/concepts/resource_types/sequences). - name: Labels - name: Relationships description: >- The relationships resource type represents connections between resource objects in CDF. Relationships allow you to organize assets in other structures in addition to the standard hierarchical asset structure. Each relationship is between a source and a target object and is defined by a relationship type and the external IDs and resource types of the source and target objects. Optionally, a relationship can be time-constrained with a start and end time. To define and manage the available relationship types, use the labels resource type. The externalId field uniquely identifies each relationship. - name: Entity matching description: >- The entity matching contextualization endpoints lets you match CDF resources. For example, you can match time series to assets. The model uses similarity between string-fields from the source and the target to find potential matches, for instance the source name and the target name. The exact algorithm may change over time. - name: Entity matching pipelines description: >- **Note** This functionality is in beta. Callers need to provide the `cdf-version: beta` header to their requests. Using Entity matching pipelines, you can configure re-runnable executions of entity matching. Pipelines support expert knowledge (confirmed and/or rejected matches), regex rules (match rules), and entity matching models. - name: Diagram parsing description: A collection of Diagram Parsing API services in Cognite Data Fusion (CDF). - name: Vision description: >- Vision API is deprecated. See [Deprecated and retired features](https://docs.cognite.com/cdf/deprecated) for details and timelines. The Vision contextualization endpoints enable extraction of information from imagery data based on their visual content. For example, you can detect external ID or name of assets, detect and read value of gauges or identify common industrial objects in images. This service has support for batch processing which enables processing of multiple image files via an asynchronous prediction request. A new contextualization job is triggered by sending a POST request to the service. The response of the POST request contains a job ID, which can then be used to make subsequent calls to check the status and retrieve the results of the job once it is completed. - description: >- Atlas AI agents use language models to solve specific industrial business needs, such as providing insights into historical and planned maintenance and analyzing time series data for root cause analysis. An agent includes prompts with instructions, industry-relevant tools, and access to industrial data stored in the Cognite Data Fusion (CDF) knowledge graph. A project can have a maximum of 30 agents. Each agent can have a maximum of 10 tools. For more information about Atlas AI agents, see the [Atlas AI documentation](https://docs.cognite.com/cdf/atlas_ai/concepts/). name: Agents - description: >- Skills are reusable instruction sets that can be shared across Atlas AI agents. Agents only load skills as needed to reduce initial context window bloat. name: Skills - description: >- Project-scoped Atlas settings, including the default orchestrator agent for incoming operator queries. name: Project Configuration - description: >- List language models available in your CDF project. Compatible with the OpenAI Models API specifications. Connect any OpenAI compatible client or SDK by pointing it to the Cognite API endpoint. name: Models - description: >- Generates chat completions using language models available in your CDF project. Compatible with the OpenAI Chat Completions API specifications. Connect any OpenAI compatible client or SDK by pointing it to the Cognite API endpoint. Feature support and model availability vary by model and project. **Rate limiting**: Requests are subject to rate limits. When a limit is exceeded, the API returns an HTTP `429 Too Many Requests` error. Retry requests using an exponential backoff strategy with jitter. name: Chat Completions - description: A collection of AI API services in CDF. name: Cognite AI - name: Documents description: >- A document is a file that has been indexed by the document search engine. Every time a file is uploaded, updated or deleted in the Files API, it will also be scheduled for processing by the document search engine. After some processing, it will be possible to search for the file in the document search API. The document search engine is able to extract content from a variety of document types, and perform classification, contextualization and other operations on the file. This extracted and derived information is made available in the form of a `Document` object. The document structure consists of a selection of derived fields, such as the `title`, `author` and `language` of the document, plus some of the original fields from the raw file. The fields from the raw file can be found in the `sourceFile` structure. The derived fields are described in more detail below. ### Derived fields #### title Some document types (such as PDFs) contain additional metadata fields. If the document contains its title as part of this metadata, this field will be populated with that title. Note that we do not currently extract the title from the document content itself. If there is a need for this, we may consider adding such functionality in the future. #### author Similar to the `title` field, the author field is another field that can often be extracted from the document's metadata. #### producer The `producer` field also exists in the document metadata. It contains information about the software or the system that was used to create the document. #### createdTime The `createdTime` we assign to the document is not exactly the same as the one found in the Files API. We first try to extract the created time from the document metadata. If the document does not contain such a timestamp, we fall back to the time set in the Files API. #### mimeType If there is a mime type set in on the file in the Files API, this field will be set to the same mime type. If there is no mime type set on the file, we will try to auto-detect it. #### extension This field contains the extension of the file, derived from the file name. For instance, if the file name is `My Document.docx`, the `extension` field will contain `docx`. #### pageCount Contains the number of pages in the document, if possible to determine. #### type The `type` field contains a high level file type, derived from the mime type. Mime types are not that pleasant to look at, and not always easy to understand. That is why we map the mime types into more user-friendly types. Below is the list of types currently returned, but be aware that this list may be extended in the future. - `Document`: Document files from Microsoft Word or similar word processing software. - `PDF`: PDF files. - `Spreadsheet`: Files from Microsoft Excel or similar spreadsheet software. - `Presentation`: Slides from Microsoft Powerpoint or similar. - `Image`: Any kind of image such as PNG or JPG files. - `Video`: Any kind of video such as MOV or MP4 files. - `Tabular data`: Csv, tsv and other kinds of tabular data files. - `Plain text`: Plain text files. - `Compressed`: ZIP files and other kinds of compressed archive files. - `Script`: Program code such as python or matlab. - `Other`: Anything that doesn't fit in any of the above types. #### geoLocation If there is a geolocation set on the file in the Files API, then this field will contain the same geolocation. If there is no explicitly assigned geolocation, the document processing system will try to detect a location using two different techniques; 1. We will extract locations from files that contain embedded GPS locations. Photos and videos often have this kind of metadata. 2. We will look at related assets that have locations, and assign the same location(s) to the document. ### File type support We create a document for each uploaded file, but only derive data from certain files. The following file types are eligible for further data extraction & enrichment: - PDF files - Spreadsheets, documents, and presentations from the Microsoft, Libre Office and macOS office suites - Plain text files - Images - name: Document preview x-displayName: Preview description: >- The document preview service is a utility API that can render most document types as an image or PDF. This can be very helpful if you want to display a preview of a file in a frontend, or for other tasks that require one of these formats. For both rendered formats there is a concept of a page. The actual meaning of a page depends on the source document. E.g. an image will always have exactly one page, while a spreadsheet will typically have one page representing each individual sheet. The document preview service can only generate preview for document sizes that do not exceed 150 MiB. Trying to preview a larger document will give an error. ### File type support Previews can be created for the following types of files: - PDF files - Spreadsheets, documents and presentations from the Microsoft and Libre Office office suites - Images - name: Geospatial description: >- The Geospatial API allows to model a problem domain when data has a geometric or geographic nature. The geospatial data is organized in feature types that are homogeneous collections of features (geospatial items), each having the same spatial representation, such as points, lines, or polygons, and a common set of typed properties. The Geospatial API is aware of Coordinate Reference Systems, and allows transformations. To learn more about geospatial concepts, see the [concept guide](https://developer.cognite.com/dev/concepts/resource_types/geospatial.html). - name: SessionsInternal - name: Sessions description: > Sessions are used to maintain access to CDF resources for an extended period of time. The methods available to extend a sessions lifetime are client credentials and token exchange. Sessions depend on the project OIDC configuration and may become invalid in the following cases - Project OIDC configuration has been updated through the [update project](#operation/updateProject) endpoint. This action invalidates all of the project's sessions. - The session was invalidated through the identity provider. - name: Extraction Pipelines description: >- Extraction Pipeline objects represent the applications and software that are deployed to ingest operational data into CDF. An extraction pipeline can consist of a number of different software components between the source system and CDF. The extraction pipeline object represents the software component that actually sends the data to CDF. Two examples are Cognite extractors and third party ETL tools such as Microsoft Azure or Informatica PowerCenter - name: Extraction Pipelines Runs description: >- Extraction Pipelines Runs are CDF objects to store statuses related to an extraction pipeline. The supported statuses are: success, failure and seen. The statuses are related to two different types of operation of the extraction pipeline. Success and failure indicate the status for a particular EP run where the EP attempts to send data to CDF. If the data is successfully posted to CDF the status of the run is ‘success’; if the run has been unsuccessful and the data is not posted to CDF, the status of the run is ‘failure’. Message can be stored to explain run status. Seen is a heartbeat status that indicates that the extraction pipeline is alive. This message is sent periodically on a schedule and indicates that the extraction pipeline is working even though data may not have been sent to CDF by the extraction pipeline. - name: Extraction Pipelines Config description: >- Extraction Pipelines Configs are configuration file revisions tied to an extraction pipeline. Users can create new configuration revisions, and extractors can fetch the latest, making it easy to deploy configuration files from source control, automated scripts, etc. - name: Extractors description: >- Extractors are tools used to move data from various source systems to CDF. The extractors API is used to manage extractor releases, give access to downloads, and contextualize which source systems each extractor is used to access. Each extractor has a list of releases, and each release may have a list of artifacts which are things like documentation PDFs, executables, and installers. Extractors may also be tied to source systems through solutions, representing concrete sources each extractor may connect to. The extractors API is currently read-only. In the future it may be possible to define per-project extractors. Because of this, cognite-maintained extractors and source systems currently have their external ID prefixed by `cognite-`. - name: Transformations description: >- Transformations enable users to use Spark SQL queries to transform data from the CDF staging area, RAW, into the CDF data model. ### Concurrency limits The number of concurrent (parallel) jobs are governed by limits. If a request exceeds one of the limits, the job fails to run. This limitation also applies to scheduled transformations. The limits for the transformation service are described in the table below. Note that under high load, some deviation from the limits might occur for short periods as the service scales up. | Limit | Per project | |-----------------------|-----------------------| | Concurrency | 10 parallel jobs | - name: Transformation Jobs description: >- Transformation jobs let you list jobs and their metrics. A maximum of 1000 jobs per transformation are retained, provided they are not older than 90 days. - name: Transformation Schedules description: >- Transformation schedules allow you to run transformations with a specific input at intervals defined by a cron expression. These transformation jobs will be asynchronous and show up in the transformation job list. Visit http://www.cronmaker.com to generate a cron expression with a UI. - name: Transformation Notifications description: >- Transformation notifications let users know when a job fails if subscribed. - name: Query description: Query lets the users preview the result of their queries. - name: Schema description: Schema provides the expected schema for CDF resources. - name: Transformation External Data Sources description: |- Manage external data source connections for Transformations. An external data source stores the credentials and location needed to reach an external storage system, and is referenced from transformation SQL by its `externalId`. Microsoft Fabric OneLake and Snowflake are the currently supported systems, selected with the `format` field. Using this API requires the relevant connector to be enabled for the project. Contact Cognite Support to have it enabled. - name: Annotations description: >- Annotations reflect contextual information in base CDF resource types, such as Files and Time series, that are not present on the object itself. The benefits of the annotations concept are threefold: - The annotations concept is a good fit for enriching the base resources themselves, so that the overall data quality is higher in a given project. - It is also a good fit for building reference datasets for data problems uniformly across customer projects. Product teams can then use those reference datasets to train machine learning models or validate the performance of their algorithms on actual customer data. - Given a uniform way of labelling similar concepts across projects, it becomes easy for apps to agree on a consistent visual representation of those concepts. - name: User profiles description: > User profiles is an authoritative source of core user profile information (email, name, job title, etc.) for principals based on data from the identity provider configured for the CDF project. User profiles are first created (usually within a few seconds) when a principal issues a request against a CDF API. We currently don't support automatic exchange of user identity information between the identity provider and CDF, but the profile data is updated regularly with the latest data from the identity provider for the principals issuing requests against a CDF API. Note: - Do not use other fields than `userIdentifier` (e.g. email) to uniquely identify a principal. User profile data is mutable and is not guaranteed to be stable or unique (except `userIdentifier`). - Do not use user profile data for authentication or authorization purposes. It is not guaranteed to be under the strict governance necessary for that. For example, one should not use email address as proof of identity or job title to give access to resources. - We strongly recommend _against_ storing any user profile data. It is intended for display purposes only and may update at any time. Clients should fetch user profile data at demand, and optionally cache for performance reasons. - name: Workflows description: >- Define and orchestrate data workflows consisting of CDF Transformations, Cognite Functions, and other processes. This service enables you to build data pipelines and business solutions leveraging the capabilities of CDF and external tools.

All API and service limitations are listed here. - name: Workflow versions - name: Workflow executions - name: Tasks - name: Workflow triggers description: >- Triggers allow you to automate the execution of your data workflows based on specific conditions, such as scheduled times (defined by cron expressions). - name: Sources description: >- A hosted extractor **source** represents an external source system on the internet. The **source** resource in CDF contains all the information the extractor needs to connect to the external source system. A source can have many jobs, each streaming different data from the source system. - name: Jobs description: >- A hosted extractor **job** represents the running extractor. Jobs produce logs and metrics that give the state of the job. For details on available states and metrics see documentation [here](https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors). - name: Destinations description: >- A hosted extractor writes to a **destination**. The destination contains credentials for CDF, and additional information about where the data should land, such as data set ID. Multiple jobs can share a single destination, in which case requests will be combined, reducing the number of requests made to CDF APIs. Metrics are still reported for each individual job. - name: Mappings description: >- A **mapping** is a custom transformation, translating the source format to a format that can be ingested into CDF. Mappings are written in the Cognite transformation language. For more details see documentation [here](https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). - name: Previews description: >- A **preview** is a temporary job that runs until it times out, fails, or receives a single message, then stores the result. This is useful for development, as it allows you to easily inspect the output of a source. Previews require a source, but not a mapping or a destination. This API is in alpha. The endpoints listed here are available only when the `cdf-version` header with the value `alpha` is provided. - name: Postgres Gateway Users description: > A postgres gateway **user** (also a typical postgres user) owns the foreign tables (built in or custom). The created postgres user only has access to use foreign tables and cannot directly create tables users. To create foreign tables use the Postgres Gateway Tables APIs - name: Postgres Gateway Tables description: View and create foreign **tables** for a given **user** - name: SAP Instances description: >- An SAP **instance** represents a configuration to an external SAP S/4HANA destination system. The **instance** resource contains all the information this API service needs to connect to an SAP S/4HANA destination. Supported SAP S/4HANA versions: * SAP S/4HANA OnPremise 2021 FPS01, and later * SAP S/4HANA Cloud - name: SAP Endpoints description: >- An SAP **endpoint** represents a configuration to an SAP S/4HANA OData endpoint (and its related OData entity) the API will send the writeback requests. It defines which SAP Instance destination and which schema mapping should be used when processing a writeback request. Supported SAP entities: * [Maintenance Notifications](https://api.sap.com/api/OP_API_MAINTNOTIFICATION/overview) * [Attachments](https://api.sap.com/api/OP_API_CV_ATTACHMENT_SRV_0001/overview) - name: Schema Mappings description: >- A **mapping** uses field and value mapping(s) to perform an in-flight transformation from source CDF entities to SAP S/4HANA entities. Mappings are written in the Cognite transformation language. For more details see the [documentation](https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors/hosted_extractors_custom_mappings). - name: Writeback Requests description: >- A writeback **request** to the SAP S/4HANA destination. The request body contains the target SAP endpoint destination, and the payload to send. - name: Simulators description: > The simulator resource contains the definitions necessary for Cognite Data Fusion (CDF) to interact with a given simulator. It serves as a central contract that allows APIs, UIs, and integrations (connectors) to utilize the same definitions when dealing with a specific simulator. Each simulator is uniquely identified and can be associated with various file extension types, model types, step fields, and unit quantities. Simulators are essential for managing data flows between CDF and external simulation tools, ensuring consistency and reliability in data handling. #### Limitations: - A project can have a maximum of 100 simulators - name: Simulator Integrations description: > The simulator integration resource represents a simulator connector in Cognite Data Fusion (CDF). It provides information about the configured connectors for a given simulator, including their status and additional details such as dataset, name, license status, connector version, simulator version, and more. This resource is essential for monitoring and managing the interactions between CDF and external simulators, ensuring proper data flow and integration. #### Limitations: - A project can have a maximum of 20 simulator integrations - name: Simulator Routines description: > The simulator routine resource defines instructions on interacting with a simulator model. A simulator routine includes: - Inputs (values set into the simulator model) - Commands (actions to be performed by the simulator) - Outputs (values read from the simulator model) Simulator routines can have multiple revisions, enabling users to track changes and evolve the routine over time. Each model can have multiple routines, each performing different objectives such as calculating optimal operation setpoints, forecasting production, benchmarking asset performance, and more. #### Limitations: - A project can have a maximum of 10000 simulator routines. - Each simulator model can have a maximum of 10 simulator routines. - Each simulator routine can have a maximum of 10 revisions. - The total size of each routine revision can be a maximum of 50.00 KB. - name: Simulator Models description: > The simulator model resource represents an asset modeled in a simulator. This asset could range from a pump or well to a complete processing facility or refinery. The simulator model is the root of its associated revisions, routines, runs, and results. The dataset assigned to a model is inherited by its children. Deleting a model also deletes all its children, thereby maintaining the integrity and hierarchy of the simulation data. Simulator model revisions track changes and updates to a simulator model over time. Each revision ensures that modifications to models are traceable and allows users to understand the evolution of a given model. #### Limitations: - A project can have a maximum of 1000 simulator models. - Each simulator model can have a maximum of 200 revisions. - name: Simulation Runs description: > Every time a simulation routine executes, a simulation run object is created. This object ensures that each execution of a routine is documented and traceable. Each run has an associated simulation data resource, which stores the inputs and outputs of a simulation run, capturing the values set into and read from the simulator model to ensure the traceability and integrity of the simulation data. Simulation runs provide a historical record of the simulations performed, allowing users to analyze and compare different runs, track changes over time, and make informed decisions based on the simulation results. When simulation run load balancing is enabled and the routine is not assigned to any particular simulator integration, the run initiates with status `queued` and any active simulator integration can claim it automatically. Without load balancing, the run initiates with status `ready` and is tied to the simulator integration specified on the routine. #### Limitations: - A retention policy is in place for simulation runs, allowing up to 100000 entries. When this limit is reached, the oldest runs are deleted to make room for new ones. - The total size of all inputs for a simulation run can be a maximum of 50.00 KB. - The total size of simulation run data sent by the connector can be a maximum of 100.00 KB. - name: Simulator Logs description: > Simulator logs track what happens during simulation runs, model parsing, and generic connector logic. They provide valuable information for monitoring, debugging, and auditing. Simulator logs capture important events, messages, and exceptions that occur during the execution of simulations, model parsing, and connector operations. They help users identify issues, diagnose problems, and gain insights into the behavior of the simulator integrations. #### Limitations: - A retention policy allows up to 10000 log entries per logId. When this limit is reached, the oldest logs are deleted to make room for new ones. - name: Units description: >- Units Catalog API provides a standardized list of units that can be used in Cognite Data Fusion. The content this API serves is based on the [CDF Units Catalog](https://github.com/cognitedata/units-catalog) - name: Unit Systems description: >- Unit system is a collection of default units for different quantities. This API provides a list of supported unit systems and their associated quantities and respective unit. - name: Application Data Storage description: '' - name: Location Filters description: '' - name: User Preferences description: '' - name: User History description: '' - name: Search Config description: '' - name: Organizations description: > An **organization** is used to group CDF projects and facilitate their management. An organization holds users, projects, and perhaps other organizations. The organization ID is what the users enter when logging into Cognite apps, such as Cognite Data Fusion. The organization has one IdP configuration, which is used for both interactive login and service account authentication against all projects in the organization. ### External identity providers (IdP) CDF supports interfacing with external IdPs to manage users and groups. The following vendors are supported: - Microsoft Entra ID (formerly known as Azure AD or Azure Active Directory) - Auth0 - Keycloak ### Users If a user can log into the external IdP configured for the organization, then they have access to the CDF organization. Which of the organization's projects they have access to, and what they may do inside those projects, is determined by the access settings within each project. After a user has logged into the organization for the first time, they will be visible in the organization's user list. Users can see each other, which enables them to collaborate on projects. ### Organization hierarchy An organization can have child organizations. The ownership relationship is materialized through the `parentId` field of the organization resource. ### Projects An organization holds CDF projects. The users that are logged into the organization can see all the projects in the organization, but what they can actually do within each project is controlled by the project's access control lists (ACLs) and other access control settings. ### Allowed clusters An organization has a list of clusters on which it can hold projects. This is the `allowedClusters` field on the resource. ### Organization admins An organization can have admins, which are identified principals that can perform an extended set of modifications on the organization, such as creating projects, changing who the admins are, and so on. Admins are identified by the `adminGroupId` field on the organization resource, which is the ID of a group that is managed in the external IdP. The different organization API endpoints have different access rules, which are documented under each endpoint. The general rule is that admins of a given organization have control over most aspects of the organization itself and full control of any sub-organizations. ### Authentication for this API Organizations are global, which means that they are not tied to specific projects or clusters. API requests against organizations are directed to `auth.cognite.com`, instead of a specific cluster and projects as for other resources. Only OAuth tokens issued by `https://auth.cognite.com` (such as the ones issued when logging into Fusion) are accepted by the organizations API. It is also possible to obtain a token by initiating a login flow against the authorization server directly. See the "Authorizations" sections for more information. - name: Applications description: >- Manage registered applications. The CDF API uses applications to enforce CORS. Applications contain a list of allowed `Origin`s that requests are checked against. The `Origin` in the request must match a pattern in at least one application registered to the project that the request is targeting. #### Access control Listing and retrieving applications requires the caller to be logged into the target organization, or to be an admin in any of its ancestor organizations. Creating, updating, and deleting applications requires the caller to be an admin of the target organization, or an admin in any of its ancestor organizations. - name: Integrations description: > Integrations represent extractors and other data pipelines that CDF monitors. Each integration tracks the tasks it runs, the errors it reports, and its liveness over time. You can create, list, retrieve, update, and delete integrations. An integration identifies the extractor type and holds the configuration, task history, and error history for that data pipeline. Tasks, actions, configuration, and errors are all scoped to an integration. Deleting an integration removes it from monitoring. After you delete it, you can't retrieve its task and error history. #### Limitations: - Up to 20 integrations can be created or updated in a single request. - Up to 100 integrations can be retrieved or deleted in a single request. - name: Integration Tasks description: > Tasks represent the discrete units of work an integration performs, such as a single configured query in a database extractor. CDF tracks each task's run history, including when it started and stopped. You can list task history or sync updates since a previous checkpoint. A task is a named unit of work in an integration, reported by the extractor. Each time a task starts or stops, CDF records a history entry so you can spot gaps or failures. A task can be one of three kinds. A scheduled task runs at set intervals or on a cron expression. A continuous task starts when the extractor starts and is expected to run until the extractor stops. A startup task runs once, before any continuous or scheduled tasks, and is typically used for initialization. The task's reported `type` is `continuous` for continuous tasks, or `batch` for scheduled and startup tasks, which both run for a finite period. Use sync rather than repeatedly listing history when you're building a live dashboard or alerting system, since it's designed for incremental polling with a cursor. #### Limitations: - Up to 100 task history entries or sync updates can be returned per request. - name: Integration Actions description: > Actions let you trigger remote operations on an integration, such as starting or stopping a task remotely. CDF tracks each action's status as the extractor picks it up and completes it. You can create actions, list or retrieve their status, and cancel pending ones. An action is a request for an integration to do something outside its normal task loop. The extractor picks up pending actions, attempts to run them, and reports the outcome to CDF. This lets you manage extractors remotely without network access to where they run. The actions available for an integration can be extractor-specific (through custom actions) or inherently provided (e.g. Start action, Stop action, Fetch Logs action) and reported by the extractor, though most extractors support starting and stopping a given task. Extractors can also define their own custom actions beyond these. You can cancel an action only while it's pending, running, or already cancel-pending. After the extractor reports completion, the outcome is final. #### Limitations: - Up to 20 actions can be created in a single request. - Up to 100 actions can be retrieved or cancelled in a single request. - name: Integration Configuration description: > Configuration lets you manage versioned settings for an integration. Each change creates a new revision, so you have a history of what was active at a given time. Extractors fetch the active revision when they start and when they report in to CDF. Integration configuration is a revisioned document associated with an integration. Every time you push a new configuration, CDF stores it as a new revision rather than overwriting the previous one. This lets you audit configuration changes and know exactly which revision an extractor was running at any point in time. Configuration revisions are immutable once created. To change behavior, create a new revision rather than editing an old one. #### Limitations: - Configuration content can be at most 100 000 characters. - Up to 100 configuration revisions can be listed per request. - name: Integration Errors description: > The errors API gives you access to errors reported by an integration's tasks. Errors are associated with the task and time range during which they occurred, so you can correlate failures with specific parts of an extractor's work. You retrieve errors by listing them, optionally filtered by task or time range. An integration error is a problem an extractor encountered while running a task and reported to CDF. Each error is linked to the task it occurred in and includes a start time, plus an end time once the extractor reports it as resolved. #### Limitations: - Up to 100 errors can be returned per request. - name: Principals description: > **Principal** is an umbrella term for **user accounts** and **service accounts**. Both entities can be uniquely identified, authenticated, and authorized in CDF. Principals are unique within an organization, and therefore also within a project in the organization. Principals can access data and create and run processes (transformations, Functions) in a CDF project. - A **user account** is associated with a **person** who wants to interact with CDF. Each user account has a user profile containing a unique user ID. - A **service account** is associated with an **application** or **process** that wants to interact with CDF, such as an extractor or Cognite Functions, rather than a person. ## Authentication for this API Requests to the Principals API are directed to `auth.cognite.com`, like for organizations. Only OAuth tokens issued by `https://auth.cognite.com` (such as the ones issued when logging into Fusion) are accepted by the Principals API. It is also possible to obtain a token by initiating a login flow against the authorization server directly. See the "Authorizations" sections for more information. ## User Accounts The Principals API lets you query user accounts in an organization, and retrieve profiles. - name: Signals description: >- A signal is a notification that something has occurred in a CDF process. Users and systems may listen to signals by creating a sink and attaching subscriptions with a filter that matches the signals they are interested in. Subscriptions tie sinks to topics. Each topic represents some CDF resource like integrations or workflows. When a signal is emitted for a resource, all subscriptions for that resource's topic are evaluated. If the signal matches the subscription filter, it is pushed to the the sink. All endpoints in this API require the `cdf-version` header to be set to `beta`. - name: Data products description: > Data products are governed, ready-for-consumption data assets derived from data domains, following data mesh principles. **Key characteristics:** - **Clear ownership**: Defined data product owners with explicit access roles and accountability - **Rich metadata**: Comprehensive data modeling views, usage terms, access permissions, and descriptive documentation - **Self-service consumption**: Discoverable and well-documented for autonomous data consumer access - **Version management**: Immutable data modeling view versions with governed schemas and updatable metadata Data products act as a governance layer around data modeling, establishing trust and reliability for business decisions through clear ownership, comprehensive metadata, and standardized access patterns. **Space ownership**: A data product owns a schema space. The data modeling views that the data product governs live in that schema space. The schema space is exclusively owned by the data product. paths: /transformations/externaldata/rotatekeys: post: tags: - Transformation External Data Sources summary: Rotate external data source keys description: >- > **Required capabilities:** `transformationsExternalDataSourcesAcl:WRITE` Rotate the RSA key pair for external data sources. The current key pair is removed and a new one is generated with the given expiry. The rotation is atomic across the request: if any item fails, no keys are rotated. Only Snowflake data sources are supported. If any item refers to a data source of another format, the whole request is rejected with a 400 and nothing is rotated. operationId: rotateExternalDataSourceKeys requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ExternalDataSourceRotateKeysItems' responses: '200': description: Response with the rotated keys. content: application/json: schema: $ref: '#/components/schemas/ExternalDataSourceRotatedKeysItems' '400': $ref: '#/components/responses/400ErrorResponse' '403': $ref: '#/components/responses/ForbiddenErrorResponse' components: schemas: ExternalDataSourceRotateKeysItems: type: object required: - items properties: items: type: array items: $ref: '#/components/schemas/RotateKeysRequest' minItems: 1 maxItems: 10 description: |- External data sources to rotate keys for. All `externalId` values must be unique within the request. Only Snowflake data sources are supported; if any item refers to a data source of another format, the whole request is rejected and no keys are rotated. description: Request to rotate the RSA key pair for Snowflake external data sources. ExternalDataSourceRotatedKeysItems: type: object required: - items properties: items: type: array items: $ref: '#/components/schemas/RotatedKeysResponse' description: Response with the rotated keys. RotateKeysRequest: type: object required: - externalId - expiryTime properties: externalId: $ref: '#/components/schemas/CogniteExternalId' expiryTime: $ref: '#/components/schemas/EpochTimestamp' description: |- Rotate the key pair for one external data source. `expiryTime` must be in the future and at most one year from now. RotatedKeysResponse: type: object required: - externalId - publicKey - expiryTime properties: externalId: $ref: '#/components/schemas/CogniteExternalId' publicKey: type: string description: The newly generated RSA public key. expiryTime: $ref: '#/components/schemas/EpochTimestamp' description: The rotated key pair for one external data source. Error: type: object required: - code - message description: Cognite API error. properties: code: type: integer description: HTTP status code. format: int32 example: 401 message: type: string description: Error message. example: Could not authenticate. missing: type: array description: List of lookup objects that do not match any results. items: type: object additionalProperties: true duplicated: type: array description: List of objects that are not unique. items: type: object additionalProperties: true CogniteExternalId: description: >- The external ID provided by the client. Must be unique for the resource type. type: string maxLength: 255 example: my.known.id EpochTimestamp: description: >- The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds. type: integer minimum: 0 format: int64 example: 1730204346000 responses: 400ErrorResponse: description: The response for a bad request. content: application/json: schema: type: object required: - error properties: error: $ref: '#/components/schemas/Error' example: error: code: 400 message: '`null` values aren''t allowed.' ForbiddenErrorResponse: description: The response for a forbidden request. content: application/json: schema: type: object required: - error properties: error: $ref: '#/components/schemas/Error' example: error: code: 403 message: 'This endpoint is not enabled for project: my-project' securitySchemes: oidc-token: type: http scheme: bearer bearerFormat: OpenID Connect or OAuth2 token description: >- Access token issued by the CDF project's configured identity provider. Access token must be an OpenID Connect token, and the project must be configured to accept OpenID Connect tokens. Use a header key of 'Authorization' with a value of 'Bearer $accesstoken'. The token can be obtained through any flow supported by the identity provider. oauth2-client-credentials: type: oauth2 description: >- Access token issued by the CDF project's configured identity provider. Access token must be an OpenID Connect token, and the project must be configured to accept OpenID Connect tokens. Use a header key of 'Authorization' with a value of 'Bearer $accesstoken'. The token can be obtained through any flow supported by the identity provider. flows: clientCredentials: tokenUrl: https://your-idps.token.url/ scopes: default: https://{cluster}.cognitedata.com/.default oauth2-open-industrial-data: type: oauth2 description: >- Auth flow for Open Industrial Data. Get your client secret from https://hub.cognite.com/open-industrial-data-211. flows: clientCredentials: tokenUrl: >- https://login.microsoftonline.com/48d5043c-cf70-4c49-881c-c638f5796997/oauth2/v2.0/token scopes: default: https://api.cognitedata.com/.default oauth2-auth-code: type: oauth2 description: >- Access token issued by the CDF project's configured identity provider. Access token must be an OpenID Connect token, and the project must be configured to accept OpenID Connect tokens. Use a header key of 'Authorization' with a value of 'Bearer $accesstoken'. The token can be obtained through any flow supported by the identity provider. flows: authorizationCode: authorizationUrl: https://your-idps.authorization.url/ tokenUrl: https://your-idps.token.url/ scopes: default: https://{cluster}.cognitedata.com/.default ````