Contextualize 360° images with the API

For manual contextualization in the 3D UI, see Contextualize 360° images.

Contextualize

The way you contextualize 360° images depends on your project type.

Data modeling projects
Legacy and hybrid projects

In data modeling projects, use the 3D jobs API to get suggestions and then create contextualization approvals:

Get text detection suggestions

Run the 3D job through the Create a job using 3D source data API endpoint.Use the Create360ContextualizationJobRequest body: set job type to ImageTextDetection and source.type to Cognite360ImageCollection.

Create contextualization approvals

To connect 360° image annotations (text and polygons) to CogniteAsset instances, use the Create 360 Image contextualization API endpoint (beta).

For asset-centric (legacy) or hybrid projects, you can contextualize 360° images in two ways:

Use your own tooling to read 360° images, run text detection, and create contextualization approvals.
Use the Vision API to get text regions from your 360° images and then create annotations with the Annotations API.

The Vision API is deprecated. See Deprecated features in CDF for details. We recommend migrating to a data modeling project.

The following steps use the Vision API for text detection and the Annotations API to create approvals.

Run text detection

Use the Vision API text detection to get text regions from your 360° images.Trigger text detection with the Cognite Python SDK:

from cognite.client.data_classes.contextualization import (
    FeatureParameters,
    TextDetectionParameters,
    VisionFeature,
)

extract_job = client.vision.extract(
    features=[VisionFeature.TEXT_DETECTION],
    file_ids=file_ids,
    parameters=FeatureParameters(
        text_detection_parameters=TextDetectionParameters(threshold=text_detection_threshold)
    ),
)

file_ids: List of CDF file IDs for your 360° images.

Match text to CogniteAssets

To link detections to core data model CogniteAssets, match the detected text to the right asset.

Exact matching

Retrieve the CogniteAssets of your interest, then match detected text to asset properties such as name or externalId using exact string match.

Fuzzy matching

Use fuzzy matching (for example, trigram similarity using Python’s difflib library) so that optical character recognition (OCR) issues still match the correct asset. Fuzzy matching helps when detected text has typos, spaces, or misread characters.

Create annotations

To contextualize 360° images with the matched text, create annotations with the Annotations API. Use the annotation type images.AssetLink (legacy) or images.InstanceLink (hybrid) in the request items → data. Each annotation links one 360° image (file) to one asset and can include the spatial region where the text was detected.Create annotations URL: https://{cluster}.cognitedata.com/api/v1/projects/{project}/annotationsExample request body (hybrid — images.InstanceLink):

{
  "items": [
    {
      "annotatedResourceId": 123456789,
      "annotationType": "images.InstanceLink",
      "data": {
        "text": "PUMP-01",
        "objectRegion": {
          "polygon": {
            "vertices": [
              { "x": 0, "y": 0.4891 },
              { "x": 0.1099, "y": 0.4947 },
              { "x": 0, "y": 0.5459 }
            ]
          }
        },
        "textRegion": {
          "xMax": 0.1099,
          "xMin": 0,
          "yMax": 0.5459,
          "yMin": 0.4891
        },
        "instanceRef": {
          "externalId": "asset-external-id",
          "space": "cdf_cdm",
          "instanceType": "node",
          "sources": [
            {
              "type": "view",
              "externalId": "CogniteAsset",
              "space": "cdf_cdm",
              "version": "v1"
            }
          ]
        }
      },
      "status": "approved",
      "annotatedResourceType": "file",
      "creatingApp": "my-app",
      "creatingAppVersion": "1.0.0",
      "creatingUser": "user@example.com"
    }
  ]
}

Data engineering

​Contextualize

Contextualize