About records and streams

Documentation Index

Fetch the complete documentation index at: https://docs.cognite.com/llms.txt

Use this file to discover all available pages before exploring further.

When you’re working with high-volume industrial data, storing each entry as a node in your knowledge graph becomes impractical. The graph becomes cluttered, query performance degrades, and you quickly hit instance budget limits. Records solve this problem by providing high-performance storage for bulk structured data that doesn’t need to be part of the graph structure. Records are structured data objects stored in streams, separate from the industrial knowledge graph. Like data modeling instances, records use containers to define their schema and spaces for access control, but they don’t create nodes or edges in the graph. This design enables you to store billions of records without impacting graph performance or consuming instance budgets. After reading this article, you’ll understand when to use records versus data modeling instances, how streams organize and manage your data, and how records integrate with your existing data modeling infrastructure.

​

When to use records

Records are designed primarily for immutable, high-volume data that doesn’t require complex graph relationships. The service is optimized for write-once, read-many scenarios where data volume and analytics are critical. Common use cases include:

• High-volume immutable data: Logs, events, and notifications (OPC UA events, PI EventFrames, well logs, manufacturing batch logs)
• Archived and historical data: Completed work orders, resolved alarms, concluded activities
• Data with a defined lifecycle: Active work orders or alarms that need updates during their lifecycle before being archived to immutable storage

To understand when to choose records versus data modeling instances, see the comparison table below for detailed differences.

​

Core concepts

To work effectively with records, you need to understand these key concepts:

• Streams define the lifecycle and performance characteristics of your data.
• Records are the individual data objects you store.
• Spaces provide access control and organization.
• Containers define the schema structure.

​

Streams

Streams are logical containers that organize your records and define how they behave throughout their lifecycle. When you create a stream, you choose a template that sets policies for:

• Retention periods: How long records are stored before automatic deletion
• Mutability: Whether records can be updated after ingestion
• Performance characteristics: Ingestion and query throughput limits

Each stream is created from one of the available stream templates, and this template cannot be changed after creating the stream. Streams are independent from spaces and containers. A single stream can contain records from multiple spaces with different container schemas, giving you flexibility in how you organize your data. Before you can ingest records, you must create a stream to hold them. When you query records, you reference the specific stream where they’re stored.

​

Stream templates

When you create a stream, select a stream template that defines the stream’s behavior, performance, and lifecycle policies. Keep in mind these important considerations:

• The choice between mutable and immutable records has significant scale implications. Different stream templates support different maximum record counts and storage capacities.
• You cannot change a stream’s template after creation, so choose your template carefully for production use.
• Review the limits and specifications for each template in the Streams API documentation before creating your streams, and select based on your expected data volume and mutability requirements.

Immutable streams These templates are for data that should not be modified after ingestion. Mutable streams These templates allow data updates but have lower total record limits compared to immutable streams. Once you create a stream with a specific template, you cannot change that template. If you need different template settings, you must delete all records from the stream, delete the stream, then create a new stream with the correct template.

​

Stream naming rules and limits

Stream externalId values must start with a lowercase letter, contain only lowercase letters, digits, hyphens, and underscores, and be at most 100 characters long. The value must match this pattern: ^[a-z]([a-z0-9_-]{0,98}[a-z0-9])?$. The number of active streams per project is limited. For current limits, see Limits and restrictions. Each stream template defines specific limits for records, storage, and throughput. If you exceed these limits, the Records API returns HTTP 429 Too Many Requests responses. For detailed specifications, limits, and throughput rates for each template, see the Streams API documentation. For a summary of Records resource limits and API operation limits, see Limits and restrictions. If you’re building applications or services that use Records, implement the recommended approaches for managing concurrency and rate limits to avoid hitting these limits.

​

Query time range limits

Immutable streams require a lastUpdatedTime range on every filter and aggregate query. The maxFilteringInterval setting on each stream template defines the maximum span between the gt (start) and lt (end) timestamps in a single request. For example, the BasicArchive template has a maxFilteringInterval of 365 days. This means each request can cover at most a 365-day window, but this window can be anywhere in the stream’s history, not just relative to the current date. If the difference between gt and lt exceeds the interval, the API returns a validation error. Since BasicArchive has unlimited data retention, all historical data remains accessible. To query data spanning more than 365 days, split your requests into adjacent time windows that each stay within the limit. The following table shows an example for a multi-year query. For mutable streams, lastUpdatedTime is optional, but using it improves query performance.

​

Pagination and data retrieval

The Records API provides three endpoints for consuming data, each with different pagination behavior:

• filter: Returns up to 1,000 records in a single response. This endpoint does not support cursor-based pagination. It’s designed for interactive queries where you need custom sorting and expect a bounded result set. If you need to retrieve more than 1,000 matching records, use the sync endpoint instead.
• sync: The only endpoint that supports cursor-based pagination. It returns up to 1,000 records per page, and you iterate through results by passing the cursor from the previous response. Use this endpoint for batch processing, data exports, or any workflow that needs to process large volumes of records. The sync endpoint provides the same filtering capabilities as filter, but does not support custom sorting.
• aggregate: Returns all results in a single response. Cursor-based pagination is not supported because aggregations compute over the entire matching dataset. The result size is inherently bounded by the aggregation structure, not by the number of individual records.

​

Deleting streams

Streams are resource-intensive, long-lived entities designed to persist for the lifetime of your project. Plan your stream strategy carefully and avoid patterns that involve repeatedly creating and deleting streams. When you delete a stream, it enters a soft-deleted state to protect against accidental data loss. Streams have no backup mechanism, so soft-delete is the only way to recover from accidental deletion. During this period:

• The stream and its data are preserved but inaccessible (no ingestion or queries).
• The stream doesn’t count toward the active stream limit.
• The stream’s externalId is reserved and cannot be reused for a new stream until the soft-delete period expires.
• You can recover the stream by contacting Cognite Support.

The duration of the soft-deleted state depends on the stream template. After this period, the stream and its data are permanently deleted, and the externalId becomes available for reuse. A single project can have a limited number of soft-deleted streams at any given time. To avoid hitting this limit, avoid creating and deleting streams frequently. For limits on active and soft-deleted streams, see Limits and restrictions.

​

Data retention

Some stream templates define a dataDeletedAfter retention period that controls how long records are kept before they are automatically removed. You can check a stream’s retention setting by retrieving the stream at /api/v1/projects/{project}/streams/{streamId} and inspecting settings.lifecycle.dataDeletedAfter. The value is an ISO 8601 duration (for example, P7D for seven days). If this field is absent, the stream has unlimited retention.

​

Records

Records are individual data objects that represent events, logs, or historical entries. Whether a record is immutable or mutable depends on the stream template you choose when creating the stream. An industrial knowledge graph describes relationships between entities using nodes and edges. Nodes can represent physical entities, such as equipment, or logical concepts, such as activities and process stages. However, when handling bulk data such as logs or historical records, storing each individual record as a node increases relational complexity and degrades query and retrieval performance. The following diagram represents this anti-pattern you should avoid. Use the Records service to avoid these performance penalties for high-volume data. Records, together with streams, let you store high-volume structured data in bulk, improving both the performance and scalability of your CDF-based solutions. Immutability is a key design feature for records that guarantees historical records cannot be altered, while also delivering cost-effective support for massive storage volumes. Although records support mutability through mutable stream templates, updating records comes at a significant processing and ingestion cost compared to data modeling instances. Use mutable streams as a transitional stage for data that needs updates during its lifecycle, then archive finalized records to an immutable stream for permanent storage.

​

Updating records in mutable streams

Records do not support partial updates. When you upsert a record in a mutable stream, you must provide the complete state of the record, including all properties for the container. The upsert operation replaces the entire record, any properties you omit are not preserved from the previous version. This means that all non-nullable properties must be included in every upsert request, even if you only want to change one field. Omitting a non-nullable property causes a validation error. The recommended workflow for updating a record is:

1. Read the existing record using the filter or sync endpoint.
2. Merge your changes into the full property set.
3. Upsert the complete record back to the stream.

​

Records vs. nodes

​

Identifiers for records

In data modeling, you identify nodes using a combination of the space ID and the mandatory node external ID. The external ID must be unique within the space it’s scoped to, but you can reuse the same external ID across different spaces. Records also use external IDs. Like data modeling nodes, a record’s external ID belongs to a space and is stored in a stream that can include records from multiple spaces. For records, the stream type determines the uniqueness constraints:

• Mutable streams: the service enforces uniqueness for each combination of external ID, space ID, and stream. When you update a record with the same external ID/space/stream combination, it updates the existing record rather than creating a new one.
• Immutable streams: the service does not enforce uniqueness. An immutable stream can contain multiple records with the same stream/space/external ID combination. This is useful for storing the full history of a record over time. You can use filtering capabilities to retrieve these records in bulk.

​

Spaces

Records use data modeling spaces for access control and organization. You must define a space before you can ingest records into it. Records can share spaces with data modeling instances, be stored in dedicated spaces, or use multiple shared spaces depending on your access control requirements. The following diagram illustrates three common space organization patterns: shared spaces where instances and records coexist, independent spaces for separate organization, and multiple shared spaces where records can belong to multiple spaces simultaneously.

​

Containers

Records use data modeling containers to define their schema. You can only ingest records into containers with usedFor set to record. Containers designated for records (usedFor: record) support significantly more properties than standard containers used for nodes and edges. See Limits and restrictions for specific property limits. Not all container capabilities apply to records. When designing containers for records, review the following tables to understand which features are supported and which are not supported.

​

Constraints and indexes

Constraints and indexes are not available for record containers.

​

Property settings and types

Some individual property settings are not supported for record containers.

​

Direct relations

Direct relation properties are supported in record containers, but with reduced validation and no auto-creation behavior.

​

Linking records to the knowledge graph

You can link records to the knowledge graph by defining direct relation properties in your record container schema. These properties enable you to contextualize records by storing references to data modeling instances using their space and external ID. For instance, you can link sensor logs to specific wells or alarm records to particular assets. To link sensor logs to a Well in your data model:

1. Define a direct relation property in your record container schema, such as well of type direct relation.
2. Assign the relationship when ingesting records by providing the space and external ID of the target instance.
3. Query and filter efficiently using this property to retrieve all records associated with a specific instance.

This approach enables the Records service to filter data efficiently without scanning entire streams. For example, you can run queries like “retrieve all sensor logs for well X” or “find all maintenance records for asset Y.” When designing your container schema, include direct relation properties for instances you’ll frequently use in filters and aggregations.

​

Capabilities

Records and streams have their own capabilities for access control. These capabilities are independent of each other and are not inherited from the data modeling service. However, because records rely on the data modeling container feature, you must have the dataModels:READ capability to read or write records in a stream.

​

Data ingestion

The Records service operates with near real-time consistency. When you ingest or update records, there is typically a brief delay, up to a few seconds, between when the API returns a successful response and when the changes become visible in search results, filters, and aggregations. This delay occurs because the service periodically makes newly ingested data searchable, balancing performance for high-volume data ingestion with quick data availability. In most cases, new or updated records become searchable within 1-2 seconds of ingestion. Keep this near real-time consistency in mind when designing your application:

• Write-then-read scenarios: If you ingest a record and immediately query for it, the record may not appear in the results yet. Consider implementing a brief retry mechanism or delay if your workflow depends on immediate read-after-write consistency.
• Immediate updates: For use cases with low data volumes requiring immediate visibility of every update, consider using data modeling instances instead of records.

​

Getting started

To begin using records and streams effectively, start by identifying your high-volume data sources and the target structure you need. Then explore:

• Get started with Records — Complete tutorial that walks you through creating schemas, setting up streams, ingesting records, querying, and building stream-to-stream pipelines.
• Aggregate records reference — Use aggregations to compute statistics and analyze trends across records without retrieving individual items.