Skip to main content
When you’re working with high-volume industrial data, storing each entry as a node in your knowledge graph becomes impractical. The graph becomes cluttered, query performance degrades, and you quickly hit instance budget limits. Records solve this problem by providing high-performance storage for bulk structured data that doesn’t need to be part of the graph structure. Records are structured data objects stored in streams, separate from the industrial knowledge graph. Like data modeling instances, records use record containers to define storage schemas, record views as consumption interfaces, and spaces for access control. Records don’t create nodes or edges in the graph, which enables you to store billions of records without affecting graph performance or consuming instance budgets. After reading this article, you’ll understand when to use records versus data modeling instances, how streams organize and manage your data, how record views separate storage from consumption, and how records integrate with the industrial knowledge graph.

​
When to use records

Records are designed primarily for immutable, high-volume data that doesn’t require complex graph relationships. The service is optimized for write-once, read-many scenarios where data volume and analytics are critical. Common use cases include:
  • High-volume immutable data: Logs, events, and notifications (OPC UA events, PI EventFrames, well logs, manufacturing batch logs)
  • Archived and historical data: Completed work orders, resolved alarms, concluded activities
  • Data with a defined lifecycle: Active work orders or alarms that need updates during their lifecycle before being archived to immutable storage
If your data requires frequent updates and will never be archived, consider using data modeling nodes instead. Nodes are better suited for persistent, frequently-changing entities in your knowledge graph.
To decide whether records fit your workload and how much stream capacity you need, see Choosing and sizing streams. To understand when to choose records versus data modeling instances, see the comparison table below for detailed differences.

​
Core concepts

To work effectively with records, you need to understand these key concepts:
  • Streams define the lifecycle and performance characteristics of your data.
  • Records are the individual data objects you store.
  • Spaces provide access control and organization.
  • Record containers define the schema structure.
  • Record views define a versioned interface for reading and writing records.

​
Streams

Streams are logical containers that organize your records and define how they behave throughout their lifecycle. When you create a stream, you choose a template that sets policies for:
  • Retention periods: How long records are stored before automatic deletion
  • Mutability: Whether records can be updated after ingestion
  • Performance characteristics: Ingestion and query throughput limits
Each stream is created from one of the available stream templates, and this template cannot be changed after creating the stream. Streams are independent from spaces and containers. A single stream can contain records from multiple spaces with different container schemas, giving you flexibility in how you organize your data. Using one stream across multiple solutions. Because containers define schemas and spaces define access boundaries, a single stream can serve many solutions at the same time. Each solution can define its own containers and spaces and write into a shared stream, while the stream itself defines lifecycle, retention, and performance tier rather than solution isolation. Plan stream assignments around lifecycle and throughput needs, not around solution boundaries. Before you can ingest records, you must create a stream to hold them. When you query records, you reference the specific stream where they’re stored.

​
Stream templates

Stream templates are in betaStream templates aren’t in production yet. For the current limit values of each template, see Stream template limits.
When you create a stream, select a stream template that defines the stream’s behavior, performance, and lifecycle policies. Keep in mind these important considerations:
  • The choice between mutable and immutable records has significant scale implications. Different stream templates support different maximum record counts and storage capacities.
  • You cannot change a stream’s template after creation, so choose your template carefully for production use.
  • Review Stream template limits before creating your streams, and select based on your expected data volume and mutability requirements.
Immutable streams These templates are for data that should not be modified after ingestion.
  • BasicArchive is intended for perpetual data storage. Records are kept until you delete the stream. However, overall data volume is limited, so plan usage accordingly.
  • ImmutableTestStream is exclusively for experimentation. It’s configured for high throughput and total data volume but has short data retention. Low retention in a soft-deleted state means you can quickly discard such streams when you no longer need them or recreate them to remove experimental data.
Note: Never use the ImmutableTestStream template for production purposes. As this template allows significant load on the system, if we detect improper usage patterns, we may change the settings of streams created from this template as a last resort.
Mutable streams These templates allow data updates but have lower total record limits compared to immutable streams.
  • BasicLiveData is intended for the production mutable working set (records that still change). It offers significant throughput but a low total record count compared to immutable templates.
For the limit values of each template — maximum record counts, data volumes, throughput, retention, and soft-delete windows — see Stream template limits. For how consumed properties accumulate and how to plan around the limit, see Max unique properties in Choosing and sizing streams. The budget is consumed per container the first time you write a property through it, so containers in different spaces count separately even when their schemas are identical. Referencing containers or views in queries doesn’t consume budget.
For enterprise customers with larger workloads, we can enable higher per-stream limits on request. This is not automatic with your subscription. For the available capacity, see On-request capacity, and for the request process, see Request additional capacity.
Once you create a stream with a specific template, you cannot change that template. If you need different template settings, you must delete all records from the stream, delete the stream, then create a new stream with the correct template.

​
Stream naming rules and limits

Stream externalId values must start with a lowercase letter, contain only lowercase letters, digits, hyphens, and underscores, and be at most 100 characters long. The value must match this pattern: ^[a-z]([a-z0-9_-]{0,98}[a-z0-9])?$. The number of active streams per project is limited. An active stream is any stream that exists and has not been deleted. Soft-deleted streams (streams that have been deleted but are still within their template’s recovery window) do not count toward this limit. For current limits, see Stream limits per project. Each stream template defines specific limits for records, storage, and throughput. For the limit values of each template, see Stream template limits. If you’re building applications or services that use Records, implement the recommended approaches for managing concurrency and rate limits to avoid hitting these limits.

​
Query time range limits

Immutable streams require a lastUpdatedTime range on every filter and aggregate query. The maxFilteringInterval setting on each stream template defines the maximum span between the gt (start) and lt (end) timestamps in a single request. The limit follows from how the two stream types store data. Immutable streams write records into time-based partitions that roll over as data arrives, with every property indexed within each partition. The lastUpdatedTime range bounds how many partitions one request touches, so query performance stays stable as a stream grows to billions of records. Mutable streams keep one continuously updated index instead, which is why they don’t need the range and why their capacity limits are much lower. For example, the BasicArchive template has a maxFilteringInterval of 365 days. This means each request can cover at most a 365-day window, but this window can be anywhere in the stream’s history, not just relative to the current date. If the difference between gt and lt exceeds the interval, the API returns a validation error. Since BasicArchive has unlimited data retention, all historical data remains accessible. To query data spanning more than 365 days, split your requests into adjacent time windows that each stay within the limit. The following table shows an example for a multi-year query.
You can check a stream’s current maxFilteringInterval by retrieving the stream at /api/v1/projects/{project}/streams/{streamId} and inspecting settings.limits.maxFilteringInterval. The value is an ISO 8601 duration (for example, P1Y for one year or P7D for seven days).
For mutable streams, lastUpdatedTime is optional, but using it improves query performance.

​
Pagination and data retrieval

The Records API provides three endpoints for consuming data, each with different pagination behavior:
  • filter: Returns up to 1,000 records in a single response. This endpoint does not support cursor-based pagination. It’s designed for interactive queries where you need custom sorting and expect a bounded result set. If you need to retrieve more than 1,000 matching records, use the sync endpoint instead.
  • sync: The only endpoint that supports cursor-based pagination. It returns up to 1,000 records per page, and you iterate through results by passing the cursor from the previous response. Use this endpoint for batch processing, data exports, or any workflow that needs to process large volumes of records. The sync endpoint provides the same filtering capabilities as filter, but does not support custom sorting.
  • aggregate: Returns all results in a single response. Cursor-based pagination is not supported because aggregations compute over the entire matching dataset. The result size is inherently bounded by the aggregation structure, not by the number of individual records.
There’s no endpoint that retrieves a single record by identifier. To fetch one record, use filter with equals on ["space"] and ["externalId"]. On an immutable stream, the same identifier can match several records, one per version.

​
Deleting streams

Streams are resource-intensive, long-lived entities designed to persist for the lifetime of your project. Plan your stream strategy carefully and avoid patterns that involve repeatedly creating and deleting streams. When you delete a stream, it enters a soft-deleted state to protect against accidental data loss. Streams have no backup mechanism, so soft-delete is the only way to recover from accidental deletion. During this period:
  • The stream and its data are preserved but inaccessible (no ingestion or queries).
  • The stream doesn’t count toward the active stream limit.
  • The stream’s externalId is reserved and cannot be reused for a new stream until the soft-delete period expires.
  • You can recover the stream by contacting Cognite Support.
The duration of the soft-deleted state depends on the stream template. After this period, the stream and its data are permanently deleted, and the externalId becomes available for reuse. A single project can have a limited number of soft-deleted streams at any given time. To avoid hitting this limit, avoid creating and deleting streams frequently.
We expect streams to be long-lived. The exception is streams created with one of the test templates. Deleting a stream can take a long time, depending on the stream settings and the volume of data stored.
For limits on active and soft-deleted streams, see Stream limits per project.

​
Data retention

Some stream templates define a dataDeletedAfter retention period that controls how long records are kept before they are automatically removed. The retention period is fixed when you create the stream and can’t be changed afterwards.
Automatic data retention only applies to immutable streams. Mutable streams do not support the dataDeletedAfter setting. If you need to manage the lifecycle of records in a mutable stream, you must handle it yourself. For example, by deleting records directly or archiving finalized records to an immutable stream with a retention policy.
You can check a stream’s retention setting by retrieving the stream at /api/v1/projects/{project}/streams/{streamId} and inspecting settings.lifecycle.dataDeletedAfter. The value is an ISO 8601 duration (for example, P7D for seven days). If this field is absent, the stream has unlimited retention.
Record deletion after the retention period is not instantaneous. Expired records are removed in large batches, and there may be a delay of up to several days between a record’s expiration and its actual deletion. During this time, expired records remain queryable and continue to count toward the stream’s storage and record count limits. Plan your stream capacity accordingly, accounting for this buffer when estimating total storage needs.

​
Records

Records are individual data objects that represent events, logs, or historical entries. Whether a record is immutable or mutable depends on the stream template you choose when creating the stream. An industrial knowledge graph describes relationships between entities using nodes and edges. Nodes can represent physical entities, such as equipment, or logical concepts, such as activities and process stages. However, when handling bulk data such as logs or historical records, storing each individual record as a node increases relational complexity and degrades query and retrieval performance. The following diagram represents this anti-pattern you should avoid. Use the Records service to avoid these performance penalties for high-volume data. Records, together with streams, let you store high-volume structured data in bulk, improving both the performance and scalability of your CDF-based solutions. Immutability is a key design feature for records that guarantees historical records cannot be altered, while also delivering cost-effective support for massive storage volumes. Although records support mutability through mutable stream templates, updating records comes at a significant processing and ingestion cost compared to data modeling instances. Use mutable streams as a transitional stage for data that needs updates during its lifecycle, then archive finalized records to an immutable stream for permanent storage.

​
Updating records in mutable streams

Records do not support partial updates. When you upsert a record in a mutable stream, you must provide the complete state of the record, including all properties for the container. The upsert operation replaces the entire record, any properties you omit are not preserved from the previous version. This means that all non-nullable properties must be included in every upsert request, even if you only want to change one field. Omitting a non-nullable property causes a validation error. The recommended workflow for updating a record is:
  1. Read the existing record using the filter or sync endpoint.
  2. Merge your changes into the full property set.
  3. Upsert the complete record back to the stream.
This differs from data modeling instances, which support partial updates where you only need to send the properties you want to change. For records, every upsert is a full replacement.

​
Records vs. nodes

​
Identifiers for records

In data modeling, you identify nodes using a combination of the space ID and the mandatory node external ID. The external ID must be unique within the space it’s scoped to, but you can reuse the same external ID across different spaces. Records also use external IDs. Like data modeling nodes, a record’s external ID belongs to a space and is stored in a stream that can include records from multiple spaces. For records, the stream type determines the uniqueness constraints:
  • Mutable streams: the service enforces uniqueness for each combination of external ID, space ID, and stream. When you update a record with the same external ID/space/stream combination, it updates the existing record rather than creating a new one.
  • Immutable streams: the service does not enforce uniqueness. An immutable stream can contain multiple records with the same stream/space/external ID combination. This is useful for storing the full history of a record over time. You can use filtering capabilities to retrieve these records in bulk.
In a single write request to a mutable stream, all combinations of space + externalId must be unique. You cannot create and update a record with the same space and external ID combination in the same POST request to /streams/{streamId}/records.

​
Spaces

Records use data modeling spaces for access control and organization. You must define a space before you can ingest records into it. Records can share spaces with data modeling instances, be stored in dedicated spaces, or use multiple shared spaces depending on your access control requirements. The following diagram illustrates three common space organization patterns: shared spaces where instances and records coexist, independent spaces for separate organization, and multiple shared spaces where records can belong to multiple spaces simultaneously.
Currently, you can delete a space that contains records. In an upcoming release, deleting a space with records will be prevented. You’ll need to delete all records from the space before you can delete the space itself, making this consistent with data modeling behavior.If you delete a space and recreate it with the same external ID, the records will still be associated with the original space ID and remain accessible in queries.

​
Record containers

Record containers are data modeling containers that define the schema for records. You can only ingest records into containers with usedFor set to record. Containers designated for records (usedFor: record) support significantly more properties than standard containers used for nodes and edges. See Limits and restrictions for specific property limits.
Deleting a container makes all records using that container permanently inaccessible, even if you recreate a container with the same external ID and schema.Plan your container schema carefully before ingesting records, as you cannot recover access to records after deleting their container.
Not all container capabilities apply to records. When designing containers for records, review the following tables to understand which features are supported and which are not supported.

​
Constraints and indexes

Constraints and indexes are not available for record containers.

​
Property settings and types

Some individual property settings are not supported for record containers.

​
Direct relations

Direct relation properties are supported in record containers, but with reduced validation and no auto-creation behavior.

​
Linking records to the knowledge graph

You can link records to the knowledge graph by defining direct relation properties in your record container schema. These properties enable you to contextualize records by storing references to data modeling instances using their space and external ID. For instance, you can link sensor logs to specific wells or alarm records to particular assets. To link sensor logs to a Well in your data model:
  1. Define a direct relation property in your record container schema, such as well of type direct relation.
  2. Assign the relationship when ingesting records by providing the space and external ID of the target instance.
  3. Query and filter efficiently using this property to retrieve all records associated with a specific instance.
This approach enables the Records service to filter data efficiently without scanning entire streams. For example, you can run queries like “retrieve all sensor logs for well X” or “find all maintenance records for asset Y.”
Validation behavior for direct relationsWhen you ingest records with direct relation properties, the Records service validates that the target space exists but does not verify that the referenced instance exists. This differs from standard data modeling behavior, where both space and instances are validated.You can ingest a record with a direct relation pointing to a non-existent instance, as long as the target space is valid. Design your ingestion pipelines to ensure referenced instances exist before creating records that link to them.
When designing your container schema, include direct relation properties for instances you’ll frequently use in filters and aggregations.

​
Traversing between records and the graph

Traversal betweeb Records and Data Modeling is a two-step query in either direction, and record views make the link discoverable from both sides.
  • Records to graph: Filter records on the records side first (for example, critical alarms from the last hour), collect the distinct { "space", "externalId" } values of the direct relation property that points to instances in the graph, and retrieve those instances from data modeling through the view the record view declares as source. Dangling references are simply absent from the result.
  • Graph to records: Filter instances on the data modeling side first (for example, all crude oil pumps), then query records with an equals filter on the direct relation property for a single instance, or an in filter with up to 100 { "space", "externalId" } values in one request. To reference more than 100 instances, split them across several requests, or combine several in clauses with or within the filter limits.

​
Record views

A record view is a named, versioned interface for reading and writing records. Record views follow the same separation of storage and consumption as data modeling views: containers define where properties are stored, while views define the property names and relationships that applications use. Use record views to keep applications independent of the physical record schema. A record view can:
  • Map properties from multiple record containers into one flat interface.
  • Rename container properties without changing stored data.
  • Associate the interface with a stream, so consumers don’t need to know which stream to use.
  • Provide one source reference for ingest, upsert, filter, sync, and aggregate operations.
  • Add type information to direct relations by identifying the data modeling view used to read the target instances.
The streamId field binds a record view to its backing stream: requests that use the view must target that stream, and clients can read the stream identifier from the view instead of hard-coding it. A record view can map multiple containers, but every mapped container must have usedFor: record. A regular data modeling view can’t map record containers, and a record view can’t map containers used for nodes or edges. For the full view schema and API operations, see Views. Use the view identifier and view property identifiers in client code. The view resolves each exposed property to its underlying container property and preserves the stream association. You can therefore evolve storage behind a new view version without coupling consumers to container identifiers.
Data models can’t include record views yet, and records aren’t returned by /models/instances/query or GraphQL. Like time series data points, records are only queryable through their own endpoints. Read and write records through the Records API endpoints, and traverse to the graph with the two-step pattern described in Traversing between records and the graph.

SensorLog references Sensor with a direct relation, and Sensor exposes the reverse direct relation back to its SensorLog records

​
Implicit container filtering

Reading through a view, with a view source or a view property path in a filter or aggregation, returns only records that have data in all containers the view maps (the view’s mappedContainers). This matches data modeling views and is what lets one stream hold several record types: a view that maps a shared container and an alarm container returns alarms only, because other record types have no data in the alarm container.

​
View-level filters

A record view can carry a filter. Every read through the view, with filter, sync, and aggregate, applies it, combined with the implicit hasData container filtering and with the filter in the request.
A consumer who queries OpenAlarms with a priority filter in the request gets unacknowledged, high-priority alarms without mentioning acknowledgement at all.

​
Implementing other record views

Record views support implements, like data modeling views: a view that implements a base record view inherits the base view’s property mappings and adds its own, so a family of views (a base Alarm view with specialized variants) doesn’t remap the shared containers. Implemented views must be record views, and all mapped containers must be usedFor: record.

​
Connect record views to data modeling views

Record views make direct relations between records and data modeling instances discoverable from both sides of the relationship. On the record view, set the mapped direct relation property’s source to the data modeling view that consumers should use to read the target instance.
On the data modeling view, define a reverse direct relation whose source and through.source reference the record view. Set through.identifier to the record view property that exposes the direct relation.
This metadata anchors records to the knowledge graph. Consumers can traverse from a record to its related sensor through the data modeling view, or discover related records from a sensor through the reverse direct relation. The record data remains in the Records service, similar to how time series data points and file contents remain in their respective services while their resources are registered in the graph. Because the two views reference each other, create them in the same POST /models/views request. The request accepts data modeling views and record views together, so neither view has to exist before the other. To follow the relation at query time, see Traversing between records and the graph.
Record views can expose mapped direct relation properties, but they can’t define edge or reverse direct relation connection properties. Define the reverse direct relation on the data modeling view instead.

​
Capabilities

Records and streams have their own capabilities for access control. These capabilities are independent of each other and are not inherited from the data modeling service. However, because records rely on the data modeling container feature, you must have the dataModels:READ capability to read or write records in a stream.
Stream selection is not access-controlled. Access control for records operates at the space level via streamrecords:write and streamrecords:read capabilities scoped to a space. However, there is currently no capability that restricts which stream a user can write into. Any user with write access to a space can choose any available stream when ingesting records.

​
Data ingestion

Records are ingested directly through the API, the SDKs, or the extractors that support streams. See Get started with Records for the supported ingestion paths. For scheduled or medium-volume sources, write from a Cognite Function or a Data Workflow task that stays within the stream’s rate limits. To move or enrich records between streams, for example to archive finalized records from a mutable stream to an immutable one, use a Data Workflow with the record stream trigger. The Records service operates with near real-time consistency. When you ingest or update records, there is typically a brief delay, up to a few seconds, between when the API returns a successful response and when the changes become visible in search results, filters, and aggregations. This delay occurs because the service periodically makes newly ingested data searchable, balancing performance for high-volume data ingestion with quick data availability. In most cases, new or updated records become searchable within 1-2 seconds of ingestion. Keep this near real-time consistency in mind when designing your application:
  • Write-then-read scenarios: If you ingest a record and immediately query for it, the record may not appear in the results yet. Consider implementing a brief retry mechanism or delay if your workflow depends on immediate read-after-write consistency.
  • Immediate updates: For use cases with low data volumes requiring immediate visibility of every update, consider using data modeling instances instead of records.

​
Unit-aware queries

Record containers support float properties with units from the CDF unit catalog. When you query records, you can request values in a different compatible unit without modifying the stored data. The records/sync, records/filter, and records/aggregate endpoints accept a top-level targetUnits parameter that converts both response values and filter inputs to your requested unit. For example, if your container stores maxPressure in bar, you can request results in pascal and write your range filters with pascal values — the service handles the conversion in both directions automatically. Set includeTyping: true to include a typing block in the response that shows the resolved unit for each property, which is useful for building dashboards that display unit labels dynamically. For the full units reference, code examples, and data normalization patterns, see Units in CDF — Integration with records.

​
Getting started

To begin using records and streams effectively, start by identifying your high-volume data sources and the target structure you need. Then explore:
  • Choose and size streams. Decide whether records fit your workload, pick a default stream template, and request additional capacity when needed.
  • Get started with Records — Complete tutorial that walks you through creating schemas, setting up streams, ingesting records, querying, and building stream-to-stream pipelines.
  • Aggregate records reference — Use aggregations to compute statistics and analyze trends across records without retrieving individual items.
  • Property paths — Reference properties through containers or record views in filters, sorts, and aggregations.
  • Units in CDF — Integration with records — Full reference and examples for unit-aware queries, filters, aggregations, and data normalization patterns.
Last modified on August 25, 2026