Documentation Index
Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
Use this file to discover all available pages before exploring further.
Before you create production streams, use this article to decide whether records are the right CDF service for your data, which default stream template to use, and when to request capacity above the project defaults.
For limit specifications on each default template, see Stream templates in About records and streams.
Evaluate whether to use records
Records and streams provide high-performance storage for bulk structured data that lives next to the industrial knowledge graph. Typical examples include logs, events, notifications, archived history, and other append-mostly rows.
Not every workload belongs in records. Use the table below to match your data to the right CDF service.
| Your data | Use this CDF service |
|---|
| Numeric, string or state measurements indexed by time, one value per tag per timestamp (sensors, PI, OPC UA DA). | Time series and datapoints |
| Entities in the physical or logical world with rich relationships and modest cardinality. | Data modeling instances |
| Binary blobs like PDFs, P&IDs, photos, 3D, and lab files in their original format. | Files |
| Append-mostly structured rows such as alarm events, OPC UA frames, batch phase records, MES logs, lab samples, audit trails, and well log curves by depth. | Records |
Plan before you create streams
Work through these planning questions before you start large-scale ingestion. Capture your answers in your project’s architecture or capacity documentation. If you can’t answer the lifecycle and volume questions, pause implementation and gather estimates first.
- Use case. What data are you storing? How is it ingested and consumed?
- Lifecycle. Are records updated or deleted after the first write?
- Retention. Do you need data forever, or for a fixed number of years?
- Volume. How many records per day, month, and year? What is the total over your retention period?
- Streams. Do you need one stream or several, for example by site, domain, or time window?
- Fit. Does the workload fit default templates, or do you need a capacity request?
Understand stream limits
A stream template has a fixed set of limits applied when you create a stream. The limits in this section reflect the current defaults. For the full specification of each template, see Stream templates.
Request additional capacity before you create the stream. Per-stream limits are fixed at creation and can’t be raised later. If a stream outgrows its template, migrate the data to a new stream with higher limits, then delete the old stream.
Mutability
Immutable streams (ImmutableTestStream, BasicArchive) don’t allow you to update or delete individual records after ingestion. They support higher total record counts and storage volume.
Mutable streams (BasicLiveData) allow update and delete. They’re sized as a small, hot working set.
The supported production pattern is one BasicLiveData stream per project for data that’s still changing, plus one or more immutable archive streams for archival-state data. When a record reaches its archival state, write it to an immutable archive and delete it from the mutable stream. See Updating records in mutable streams.
Max records and max data volume
Together these cap the size of one stream. Typical industrial records are roughly 600 to 900 bytes (alarms, batch records, and event frames often fall in this range). Record count usually reaches the limit before storage volume. Plan against whichever binds first for your average record size.
For example, a site ingesting 7.3 million alarm records per year (~900 bytes each, ~6.1 GB/year) hits the 50 million record cap on a BasicArchive stream in about seven years. At that ingest rate, the 50 GB volume cap would take longer and record count would bind first.
Every project starts with the default templates: BasicLiveData caps at 5 million records per stream and BasicArchive at 50 million. Cognite can enable more capacity on request, up to billions of records or terabytes of data per stream. See Stream templates for default and on-request limits.
Retention
Retention (dataDeletedAfter) controls when records are automatically hard-deleted. The defaults differ by template:
| Template | Retention |
|---|
BasicArchive | Unlimited. Records are kept until you delete the stream. |
ImmutableTestStream | 7 days. |
BasicLiveData | No time-based retention. The 5 million record cap bounds mutable storage. |
Throughput
Throughput is enforced per stream in 10-minute sliding windows. There are separate limits for records ingested, bytes ingested, bytes read, and updates and deletes (mutable streams only).
For example, backfilling 45 GB at the BasicArchive ingest limit of 170 MB per 10 minutes takes about 44 hours of continuous ingestion (two days). Keep sustained load below 70% of the ceiling. Bursts and retries can throttle you if you run near the limit continuously.
Max range filter interval for lastUpdatedTime
Immutable streams require a lastUpdatedTime range on every filter and aggregate request. Each template sets maxFilteringInterval, which is the largest window between gt and lt in one request. For example, the value is 365 days on BasicArchive and 7 days on ImmutableTestStream. To query a wider span, split your requests into adjacent time windows. See Query time range limits.
For mutable streams, lastUpdatedTime is optional on queries. Using it improves query performance.
Max unique properties
The max unique properties limit is the count of distinct property names with data across the entire stream, not per record. Every property that’s ever been written through a container counts toward the limit. If you have three containers, each with 200 properties, the stream uses up to 600 unique properties.
If you approach the limit, review your schema layout or group rarely queried fields in a JSON property. If you still exceed the limit, contact your Cognite representative to discuss your use case.
The consumed property count does not decrease when you stop writing a property. On immutable streams, you also can’t delete individual records to free slots.
There’s no API today to report how many unique properties a stream has consumed. Track the properties defined on the containers that write into each stream as part of your capacity planning. Soft-delete retention
When you delete a stream, it enters a soft-deleted state. You can request recovery through Cognite Support within a template-specific window: 1 day for ImmutableTestStream and 6 weeks for production templates. Restoration is manual and typically takes about 2 weeks after you submit the request, so contact Support as soon as you notice the deletion. Don’t rely on soft-delete as a backup strategy. Projects also limit how many soft-deleted streams you can have at once. See Limits and restrictions.
Active streams per project
| Limit | Default |
|---|
| Total active streams per project | 3 |
Active streams from BasicLiveData template | 1 |
Active streams from BasicArchive template | 2 |
Active streams from ImmutableTestStream template | 3 |
Sizing example: multi-site alarms
A specialty chemical company consolidates DCS alarms from 10 plants:
| Input | Value |
|---|
| Records per year | 7.3 M/plant x 10 = 73 M/year |
| Average record size | ~900 B, or ~66 GB/year |
| Retention | 5 years |
| Backfill | 5 years (365 M records, ~330 GB) in 30 days, then steady state |
| Check | Result |
|---|
Fits BasicArchive alone? | No. 365 M exceeds the 50 M record cap and 330 GB exceeds the 50 GB volume cap. |
| Steady-state ingest | ~1,390 records per 10 minutes, well below the 170,000 limit. |
| Backfill ingest | ~76 MB per 10 minutes, under half of the 170 MB limit. |
| Stream | Configuration | Fits defaults? |
|---|
| All-plant alarm archive | Immutable, ~365 M records, ~330 GB, 5-year retention | No. The volume exceeds BasicArchive defaults. Request additional capacity with headroom above the 5-year projection. |
| Active alarm acknowledgements | Mutable, default BasicLiveData capacity | Yes. No capacity request needed. |
| Single-plant batch pilot | Immutable, ~10 M records, ~10 GB, 5-year retention | Volume fits BasicArchive, but the 5-year retention is non-default. Request additional capacity to set the retention with headroom. |
Choose a default template
CDF projects ship with three default stream templates:
| Template | Use for |
|---|
BasicArchive | Append-only history at single-facility scale. |
BasicLiveData | The production mutable working set for records that still change. |
ImmutableTestStream | Short experiments only. 7-day retention. Never use for production. |
| If… | Choose |
|---|
| Records are never updated or deleted after ingestion in production. | BasicArchive |
| Only a small subset changes until each record reaches its archival state, then history is append-only. | BasicLiveData for the working set plus BasicArchive for the archive, with an archival flow. |
| Your pipeline upserts or updates records from RAW, Functions, or extractors. | BasicLiveData, or change the pipeline to append-only before using BasicArchive. |
| You need more than 5 million records in one mutable stream. | Request additional capacity before go-live. |
| You need more than 50 million records in one immutable stream, or a non-default retention. | Request additional capacity before go-live. |
| You’re running a short experiment, benchmark, or schema iteration. | ImmutableTestStream |
Immutable streams don’t support updating or deleting individual records. If you switch from BasicLiveData to BasicArchive to gain capacity, pipelines that upsert into the stream fail until you change them to append-only ingestion.
Request additional capacity
Submit a capacity request if your sizing exceeds default per-stream limits (50 M records or 50 GB immutable, 5 M records or 15 GB mutable), you need more than 3 active streams, or you need a retention period the defaults don’t provide.
Describe what you need in record count, storage volume in GB, and retention per stream. For the upper range we can provision on request, see the table in Stream templates.
If you chose the wrong template or outgrow a stream, migrate the data to another stream with sufficient capacity and then delete the old stream. See Understand stream limits for why limits can’t be raised after creation.
To submit a request:
- If you have a Cognite Customer Success or Solution Architect contact, send the request to them.
- Otherwise, open a Cognite Support request and mention Records / Streams custom capacity.
Routine requests typically take 1 to 2 weeks. Requests that need a new limit configuration may take longer.
Next steps
