Time series in data modeling

Cognite's core data model has a built-in CogniteTimeSeries concept to ingest and retrieve time series. CogniteTimeSeries integrates with the Time Series API. To ingest and retrieve data points, you must use the same API.

This article describes how time series are handled differently in data modeling and in asset-centric data models.

Create time series

In data modeling, each time series is a node in the property graph. You can create, update, and delete time series using the same API as other instances. The instanceType is node, and the combination of space and externalId identifies the time series and links it to the data points.

Time series must be of a specific type: numeric or string. You set the type when creating the time series. The type affects how the data points are stored, and to prevent potential data loss, you can't change it later. The isStep property determines how to interpolate between data points, and you can change the property later. Data modeling doesn't support all the time series properties of the asset-centric data model.

Create time series with the API

To create time series in data modeling, use the create or update nodes/edges endpoint. The node must have data in the cdf_cdm:CogniteTimeSeries/v1 system view to be recognized as a time series. The type is the only mandatory property.

Example request

{
  "items": [
    {
      "space": "test",
      "externalId": "minimal_example",
      "instanceType": "node",
      "sources": [
        {
          "source": {
            "type": "view",
            "space": "cdf_cdm",
            "externalId": "CogniteTimeSeries",
            "version": "v1"
          },
          "properties": {
            "name": "Time series A",
            "type": "numeric"
          }
        }
      ]
    }
  ]
}

Example request with all properties

{
  "items": [
    {
      "space": "test",
      "externalId": "hello",
      "instanceType": "node",
      "sources": [
        {
          "source": {
            "type": "view",
            "space": "cdf_cdm",
            "externalId": "CogniteTimeSeries",
            "version": "v1"
          },
          "properties": {
            "type": "numeric",
            "isStep": true,
            "unit": {
              "space": "cdf_cdm_units",
              "externalId": "temperature:deg_c"
            },
            "sourceUnit": "C",

            "name": "Hello, world!",
            "description": "My new description",
            "tags": ["TagA", "TagB"],
            "aliases": ["hello_world"],

            "assets": [],
            "equipment": [],

            "sourceId": "examples",
            "sourceContext": "documentation",
            "sourceCreatedTime": "2024-08-28T13:16:25.228Z",
            "sourceUpdatedTime": "2024-08-28T13:20:25.228Z",
            "sourceCreatedUser": "John Doe",
            "sourceUpdatedUser": "Jane Doe"
          }
        }
      ]
    }
  ]
}

Create time series with the Python SDK

from cognite.client import CogniteClient
from cognite.client.data_classes.data_modeling import DirectRelationReference, NodeId
from cognite.client.data_classes.data_modeling.cdm.v1 import CogniteTimeSeriesApply

# Instantiate Cognite SDK client:
client = CogniteClient()

# Insert a new CogniteTimeSeries
client.data_modeling.instances.apply(
    CogniteTimeSeriesApply(
        space="test",
        external_id="hello",
        name="Hello, world!",
        is_step=True,
        time_series_type="numeric",
        unit=DirectRelationReference("cdf_cdm_units", "temperature:deg_c"),
    )
)

# Insert data points
client.time_series.data.insert(
    instance_id=NodeId("test", "hello"),
    datapoints=[
        (1724845953621, 0.0),
        (1724845970101, 1.0),
    ],
)

# Retrieve data points
client.time_series.data.retrieve(
    instance_id=NodeId("test", "hello"),
    start=0,
)

For more examples using the Python SDK, please see the cognite-sdk documentation.

Synchronization from data modeling to the Time Series API

When you create a CogniteTimeSeries in data modeling, it's automatically synchronized with the Time Series API (/timeseries).

In the Time Series API, the time series you created in data modeling has an instanceId property (reusing the externalId property could cause conflicts with existing time series):

{
  "instanceId": {"space": "...", "externalId": "..."},
  // other time series properties:
  "id": ...,
  "externalId": ...,
  "name": ...,
  "metadata": ...,
}

Updates

The properties you can set in data modeling, can only be updated using the Data Modeling API.

You can still update the other properties through the Time Series API:

• externalId
• assetId
• dataSetId
• metadata

You can update a time series created in data modeling to also have an externalId property in addition to the instanceId. This can be useful in scenarios where some applications understand externalId, but not instanceId.

Synchronization process

A background "sync worker" ensures that time series created in data modeling is synchronized with the same time series seen using the Time Series API.

Expected latency

This synchronization process typically happens within seconds, but isn't instantaneous/atomic. During periods of high load, the process may take a few minutes.

If you create a time series in data modeling and immediately query the Time Series API, the newly created time series may not be visible yet.

Immediate consistency guarantees

If the Time Series API receives an instanceId that it doesn't recognize, it triggers a synchronization for the instance to guarantee that the time series has been synced at least once, though it may not always be fully up to date.

A delay can happen when you update an existing CogniteTimeSeries and, for example, change the unit property. The property may not be immediately updated in the Time Series API.

You can create a time series in data modeling and immediately ingest data points into it. If the background syncer has not yet processed the new time series, the ingestion request triggers a synchronization, ensuring that the time series exists before proceeding.

Deleting CogniteTimeSeries

Time series created in data modeling can only be deleted using the data modeling API.

Some time after the time series has been deleted from data modeling (typically within seconds), it is deleted in the Time Series API.

If data points are written to the Time Series API after the instance has been deleted by the data modeling service, the data points may appear to be stored (write requests receive a 200 OK response) until the time series in the Time Series API has been deleted. After this time, all data will be deleted.

This may cause problems if you delete a time series instance and then recreate it. When a recreated time series uses the same instanceId as a deleted instance, data points for the newly created instance may get routed to the old time series. As a result, newly written data points may be missing from the new instance since they were deleted as part of the old time series deletion.

When you have to delete and recreate a time series instance, always include a verification step.

Example: Periodically send a POST timeseries/byids request to the Time Series API for the specific instanceId you have deleted, with the ignoreUnknownIds parameter set.

When the Time Series API returns an empty response, it confirms that the instance has been deleted, and you can safely recreate the time series instance.

For more information about the the /byids API endpoint, see the Time Series API documentation.

informações

This information applies to time series created using the data modeling API. It doesn't apply to time series created using the Time Series API.

Access control

Data modeling uses spaces for governance and access control, not data sets.

To create and read CogniteTimeSeries, you need the following capabilities:

• dataModelsAcl.READ to the cdf_cdm space to read the core data model schema.
• dataModelsAcl.READ to the cdf_extraction_extensions space to read properties set by Cognite extractors.
• dataModelInstancesAcl.READ to the cdf_cdm_units space to use units.
• dataModelInstancesAcl.READ/WRITE/WRITE_PROPERTIES to the space with the time series instances.

If you have read/write access to a time series, you can also read/write its data points.

If you modify dataSetId or assetId using the Time Series API, the time series and data points may be accessible to users without data modeling access. Users with access to all time series will also have access to time series in data modeling.

Time series with a write-protected data set denies write access to data points, unless the user has datasets:owner access to the data set. Properties set by data modeling aren't write-protected.

observação

Time Series with an instanceId cannot have security categories.