Skip to main content

Pagination

Most resource types in Cognite Data Fusion (CDF) can be paginated, as indicated by the nextCursor field in the response. Pass the value of nextCursor as the cursor to get the next page of limit results. Note that all parameters except cursor have to stay the same.

A request can return fewer results than its limits, as long as there is a nextCursor that indicates that there are more data to fetch.

Simple cursor example

HTTP GET /assets?limit=1
{
"nextCursor": "8ZiApWzGe5RnTAE1N5SABLDNv7GKkUGiVUyUjzNsDvM",
"items": [
{
"name": "23-TE-96116-04",
"parentId": 3117826349444493,
"description": "VRD - PH 1STSTGGEAR THRUST BRG OUT",
"metadata": {
"ELC_STATUS_ID": "1211",
"RES_ID": "525283"
},
"id": 702630644612,
"createdTime": 0,
"lastUpdatedTime": 0,
"rootId": 6687602007296940
}
]
}

Getting the next page of assets.

note

When you send additional parameters to /assets, make sure that you also include them in all subsequent requests in addition to the cursor parameter.

HTTP GET /assets?limit=1&cursor=8ZiApWzGe5RnTAE1N5SABLDNv7GKkUGiVUyUjzNsDvM

Parallel retrieval

If you download a large number of resources (for example, events with a particular filter), it can take a long time to paginate through millions of records. CDF supports parallel retrieval through the partition parameter. The parameter has the format m/n, where n is the number of partitions to split the entire data set into.

Example

This example downloads an entire data set in parallel by splitting it into 10 partitions and doing the following requests with m running from 1 to 10:

HTTP GET /events?type=WORKORDER&partition=1/10
HTTP GET /events?type=WORKORDER&partition=2/10
HTTP GET /events?type=WORKORDER&partition=3/10

Each of the results looks similar to this:

{
"items": [
{
"startTime": 1266130800000,
"endTime": 1284328234000,
"type": "Workorder",
"subtype": "VAL",
"assetIds": [4650652196144007],
"source": "cdf-tenant",
"id": 206950949774808,
"lastUpdatedTime": 1539151642457,
"createdTime": 1533747616899
}
],
"nextCursor": "bU-qc7X1jKMUbXgOYdONu2kRvpsoc60v9qh0Votm4MT0LaH0J0VVhVLoorSh8j_j"
}

Follow the nextCursor in each response until each partition is drained.

note

Make sure that you send the filter parameters to each subsequent request.

Datapoint pagination

Datapoints, including aggregates, can use pagination similar to other endpoints. The first datapoint on the second page will follow immediately after the last datapoint on the first page.

The main difference to other endpoints is that you will receive a nextCursor for each time series that has more data and that the nextCursor fields are inside the elements of the items array. In subsequent requests, you must combine each cursor alongside its corresponding time series.

If the other parameters are equal, you may combine (time series, cursor) pairs from different requests.

You may also drop the time series you have completed and increase the limit on the remaining time series. Requests with smaller time series with a higher limit will generally be more effective than paging.

Example

{
"items": [
{
"id": 154872395793457,
"externalId": "thermometer",
"isString": false,
"isStep": false,
"unit": "kelvin",
"datapoints": [
{
"timestamp": 427629600000,
"value": 183.95
}
],
"nextCursor": "jkewklavngoatOPari4nsLGNKd453sa-asl"
},
{
"id": 1897391123,
"externalId": "CO2",
"isString": false,
"isStep": false,
"unit": "ppm",
"datapoints": [
{
"timestamp": 1654027200000,
"average": 420.99
}
],
"nextCursor": "awnl2ao42aiNawKa24Lk4vab021lnkaj"
}
]
}

Now, let's take a look at the example request. For each time series, we include the corresponding cursor.

HTTP POST /timeseries/data/list
{
"items": [
{
"externalId": "thermometerA",
"cursor": "jkewklavngoatOPari4nsLGNKd453sa-asl"
},
{
"externalId": "CO2",
"cursor": "awnl2ao42aiNawKa24Lk4vab021lnkaj",
"aggregates": [
"average"
],
"granularity": "1d"
}
],
"start": 400000000000,
"end": 1700000000000,
"limit": 1
}
Click here to view the full example.

All requests are sent to this endpoint:

HTTP POST /timeseries/data/list

Regular query

Query for normal datapoints without cursors:

{
"items": [
{
"externalId": "thermometerA"
},
{
"externalId": "thermometerB"
}
],
"start": 400000000000,
"end": 1700000000000,
"limit": 1
}

Response from the first query:

{
"items": [
{
"id": 154872395793457,
"externalId": "thermometerA",
"isString": false,
"isStep": false,
"unit": "kelvin",
"datapoints": [
{
"timestamp": 427629600000,
"value": 183.95
}
],
"nextCursor": "jkewklavngoatOPari4nsLGNKd453sa-asl"
},
{
"id": 254872395793458,
"externalId": "thermometerB",
"isString": false,
"isStep": false,
"unit": "kelvin",
"datapoints": [
{
"timestamp": 427629600000,
"value": 183.95
}
]
}
]
}
  • thermometerA may have more datapoints in the range. Use the cursor to retrieve them.
  • thermometerB has reached the latest datapoint in the range. Increase start/end to retrieve more datapoints.

Aggregate query

{
"items": [
{
"externalId": "CO2-A",
"aggregates": [
"average"
],
"granularity": "1d"
},
{
"externalId": "CO2-B",
"aggregates": [
"average"
],
"granularity": "1d"
}
],
"start": 400000000000,
"end": 1700000000000,
"limit": 1
}

Aggregate response:

{
"items": [
{
"id": 1897391123,
"externalId": "CO2-A",
"isString": false,
"isStep": false,
"unit": "ppm",
"datapoints": [
{
"timestamp": 1654027200000,
"average": 420.99
}
],
"nextCursor": "awnl2ao42aiNawKa24Lk4vab021lnkaj"
},
{
"id": 2897391124,
"externalId": "CO2-B",
"isString": false,
"isStep": false,
"unit": "ppm",
"datapoints": [
{
"timestamp": 1654027200000,
"average": 420.99
}
]
}
]
}
  • CO2-A have more aggregate datapoints in the range. Use the cursor to retrieve them.
  • CO2-B has reached the latest interval in the range with at least one aggregate. Increase start/end to fetch more aggregates.

Request with cursors

Each externalId is paired with the corresponding cursor. We request time series from the first and second requests because the parameters are compatible. In this case, the start and end are equal.

HTTP POST /timeseries/data/list
{
"items": [
{
"externalId": "thermometerA",
"cursor": "jkewklavngoatOPari4nsLGNKd453sa-asl"
},
{
"externalId": "CO2-A",
"cursor": "awnl2ao42aiNawKa24Lk4vab021lnkaj",
"aggregates": [
"average"
],
"granularity": "1d"
}
],
"start": 400000000000,
"end": 1700000000000,
"limit": 1
}

Response from request with cursors:

{
"items": [
{
"id": 154872395793457,
"externalId": "thermometerA",
"isString": false,
"isStep": false,
"unit": "kelvin",
"datapoints": [
{
"timestamp": 1263769200000,
"value": 249.15
}
],
"nextCursor": "NIaoitj4jwlkrafjalkdjfalgkjladkjflgawa"
},
{
"id": 1897391123,
"externalId": "CO2-A",
"isString": false,
"isStep": false,
"unit": "ppm",
"datapoints": [
{
"timestamp": 1656626400000,
"average": 420.95
}
]
}
]
}