Enhancements
-
Cognite Charts is now an integrated part of the CDF portal under Explore and Build. New functionality includes:
- View CDF events and time series simultaneously on the chart plot.
- Set thresholds to check if time series cross a specified value.
- See calculation status messages in the user interface.
- Get better results if you’re searching for documents with Japanese content.
- Time series descriptions automatically update if you change the description in the
Descriptionparameter for the time series in the CDF API.
-
Use the new search and view options in the Data explorer to:
- Search across all data types and add filters for metadata.
- Search for content inside documents.
- Customize and sort columns for the displayed results.
- Inspect equipment surroundings and find nearby equipment by looking up the location of assets in contextualized 3D models. Use the 3D availability column on the asset panel in the asset search results view, or navigate to relevant 3D models from the asset details panel.
- Add multiple 3D models to the same 3D view to combine and compare models. You can add CAD models and point clouds/laser scans.
- Measure distances and slice 3D models to focus on specific parts using the new menu options in the 3D view.
-
We’re introducing a Data Catalog, a collection of data sets that serves as a central and secure library of data assets you can search, explore, and manage in CDF.
Use the Data Catalog to get an overview of the data that’s available to you. You can also use it:
- As part of your onboarding workflows: to get overviews of existing data and to verify that newly onboarded data appears as expected.
- As an entry point to searching and exploring data sets and data resources.
- As a way to manage data governance and user access to data resources.
- On the Extract data, you’ll find source system logos and the best-fitted data extraction tool for ingesting data into CDF. On this page, you’ll also find new Cognite extractors you can use to extract well data from source systems. In the data extraction documentation section, you’ll find articles on how to set up and run the general CDF extractors and the specific CDF extractors for well data.
- Integrate and view 360 degree images in Cognite Reveal.
- The 3D service can now process Navisworks (NWD) model files. See Supported 3D file formats.
-
We’ve added paginating to the Retrieve data points endpoint. You’ll receive a
nextCursorfor time series with multiple data points. The first data point on the second page follows immediately after the last data point on the first page. We’ve also improved performance for retrieving data points for time series using the Cognite Python SDK v5.
Images contain essential information required for optimizing operations. This release of Cognite Data Fusion (CDF) introduces image and video management, where you can upload, organize, and explore the image and video data and add annotations to the images. With the new image contextualization API, you can detect assets tags, text and people using industry-relevant detection models.We’ve also made it easier to work with the data flow from a source system into CDF with the option to create, edit, and troubleshoot the extractors’ configuration files in the CDF user interface.Read more about this and other improvements below.
Image and video management
Images contain information such as the time or location of a photo, a number on a tag, or rust on a pipe. Use the new Image and video management tool to upload and enrich images and videos to CDF. Then, organize and explore your images and videos in a list view, gallery view, or on a map and add or review annotations to the images. Read moreUsing the Vision API and the Python and JavaScript SDKs, you can find data based on visual content and link images to assets by analyzing the images with detection models, such as asset tag detection, text detection, and person detection. The person detection model protects privacy and complies with GDPR.Create and edit extractor configuration files in the CDF user interface
The Cognite extractors connect to source systems and push data to CDF. When you set up or want to change the extractors, you can now create and edit the configuration files to fit your requirements on the Extraction pipeline page. This will reduce the time and effort spent accessing the configuration file on the extractor’s host machine and make it easier to iterate and deploy the configurations settings and investigate extractor run issues.When you’re ready to apply the configuration to a production environment, we recommend setting up remote configuration files stored in the cloud using versioned files and continuous integration systems, such as GitHub Actions, or directly with the CDF API. Read more.Point cloud visualization in 3D models
With improved point cloud visualization in the 3D digital twin models, you’ll find the rendering of objects clearer, more uniformed and with higher fidelity. This will make it easier to distinguish which object is in front and which is in the back when looking at digital twins.Place measurements in 3D models
Use the Cognite Reveal SDK to measure simple point to point distances in 3D models by selecting to- and from-points. For example to fit a new piece of equipment in a designated location. The default measurement unit is meters, but the tool provides callbacks to allow configuring the unit.Historic time series
The CDF time series service now allows data points starting on 01.01.1900T00:00:00Z (1 January 1900), specified as milliseconds since the epoch. This means you can import a more complete time series history to CDF. Time should be specified as a negative integer value for data points prior to midnight on Jan 01, 1970, and as positive integer values for data points after.Cognite Data Fusion (CDF) makes data do more. This release focuses on rapid, reusable, and scalable industrial data science by introducing new code and non-code solutions. With Cognite Functions, you can deploy and host Python code for additional calculations and models using the data in CDF. Cognite Charts offers a guided user interface to perform industrial data analytics, troubleshooting, and root cause analysis.Read more about this, how we’ve polished up the CDF user interface, and other improvements below.
Cognite Charts
Cognite Charts is a graphical user interface that lets domain experts explore data and trends and perform troubleshooting and root cause analysis. No programming skills are required to use Cognite Charts. Read more.Cognite Functions
Cognite Functions offers Python functions as a service, reducing the effort and expertise needed to deploy solutions using CDF data. Model data in your programming language and according to your company or industry standards, then call the code on demand or schedule regular runs. Read more.You can deploy Cognite Functions using the Cognite API, the Cognite Python SDK, and the CDF user interface.Improved Transform data page
CDF Transformations has a new user interface to guide you through the process of transforming data from the CDF staging area to the CDF data model. The new interface is fully backward compatible with existing transformations. Read more.Improved Extract data page
The improved Extract data page makes it easy to find your best fit extractor and the installation files. You’ll also find links to the Cognite Python and .NET utils packages and SDKs to get started creating a custom extractor. Read more.Improved Explore data page
- Files are grouped by their document type on the Files tab.
- The search bar is moved to the top of the page and displays the search results count on resource types.
- You can reset your filters, and a red badge indicates that filters are applied. You can also collapse the More filters section.
- We’ve improved the navigation when you search for a tag in an engineering diagram, such as a P&ID.
New Annotations API
CDF can detect references to tags and assets in engineering diagrams, documents, and images. The detections are often called annotations. The new Annotations API introduces strict schema validation to avoid invalid annotations and a well-defined flow for suggesting, accepting, and rejecting annotations. The API also enables third-party applications to enrich documents in CDF with custom annotations. Read more.Ensure geospatial data integrity using data sets
The geospatial endpoint is the entry point for resource types enabling storing and querying item collections with a geospatial nature. You can now use data sets to document and track data lineage for the geospatial features and feature types. Read more.3D models
You can now upload 3D models in IFC open file format to enable CDF to process and host Building Information Models (BIM).June, 2022
This June release has several improvements for the end-user and the developer.View more details in complex CAD models
To see more details and better understand the facility, we’ve improved the visual quality when you view large and complex 3D models.Also, we have improved the 3D processing pipeline and the load performance so that you can load geometry while the camera is moving. Note! To take full advantage of the improvements, you need to re-upload any 3D models processed before March 2022.A better user experience signing in to CDF
Signing in to the CDF user interface can be cumbersome if you don’t know the authentication setup in your organization. With the new CDF sign-in flow, you only need to enter your organization’s domain name and specify which CDF project you are signing in to. Your organization domain name is usually your company name spelled without spaces or an abbreviation of the name.Add your customer logo to the sign-in dialogs for a more personalized experience.Authenticate your CDF solutions
To allow the developer to focus on solution development instead of setting up authentication infrastructure, it’s now possible to authenticate for solutions built on top of CDF without writing multiple lines of code. The @cognite/auth-wrapper is an OpenID Connect/OAuth 2.0 wrapper library written in JavaScript. It helps you retrieve an access token from any IdP that meets the OpenID standard, making it easy to sign in with our JavaScript SDK. Features in the authentication wrapper SDK include:- NPM supported auth package
- Support for multiple auth flows
- Detailed documentation on how to use the package
- Detailed documentation on how to navigate key IdPs (i.e., Azure AD)
Search, filter, and sort data in the CDF Data explorer
We’ve polished up the CDF Data explorer to make it easier to find the data you are looking for:- You can reuse any Search and Filters you have defined. Simply copy the URL and share it with your co-workers.
- The search criteria you enter in the Search bar are now kept when you toggle between the different tabs in the explorer.
- On the Files > File details tab, you can scroll up and down the list of metadata to find what you’re looking for.
- We’ve added a notification to let you know that you cannot upload files greater than 5GB.
- You can now sort in the metadata tables.
- We’ve fixed reported bugs and general styling issues.
Increased number of columns for sequences
Sequences now support up to 400 numeric columns (up from 200) and 200 string columns.DirectQuery support for Power BI (Early adopter)
DirectQuery is a feature in Power BI that enables you to fetch data on demand and build Power BI dashboards with hundreds of time series and thousands of data points. With DirectQuery, no data is imported or copied into Power BI Desktop. Instead, Power BI queries CDF for the data every time you interact with the dashboard.Cognite Data Source for Grafana
The Cognite Data Source for Grafana now supports the CDF relationships resource type. You can fetch time series based on relationships in the Time series from asset tab and filter the relationships on data set, labels, and active relationships based on the time filter selected in Grafana. The new Relationships tab allows you to use Grafana’s Node Graph visualization plugin or the new (alpha) custom visualization plugin available at cognite-grafana-relationships-visualization.Templates support (Early adopter)
We’ve added a new Templates tab to support the CDF Templates feature. Using templates with Grafana allows you to scale your dashboards dynamically when you add equipment. You can enable this feature in the data source settings.April, 2022
Data governance is key to ensuring high quality throughout your data’s lifecycle, and Cognite Data Fusion (CDF) has the tools and features to secure that the data meet your expectations. This release optimizes data management by introducing more granular access to data transformations using data sets.Read more about this and the other improvements for the CDF April 2022 release below.Scope transformation access to data sets
Protect access to CDF SQL transformations from ungoverned actions and visibility by associating a transformation to a data set. Users will only be able to see and run the transformations associated with the data sets they have access to.-
As a project admin granting users access to transformations, select the data sets a user can see and run transformations for under Scope in the Transformations capability dialog:
-
As a user creating a transformation, select the data set you want to associate to a transformation in the Create new transformation dialog. Only users with transformation access to this data set can see and run the transformation.
Ingest data into sequence rows
You’ll now find Sequence Rows as an option in the Destination type field on the Transformations page in the CDF user interface and no longer need to use other developer tools like APIs and SDKs to ingest sequence rows into the Sequence resource type in CDF.Develop applications using Java
Already, you can build solutions using Cognite’s SDKs. Now, we introduce the Cognite Java SDK to interact with the CDF v1 APIs. The Java SDK is optimized for data publishing and allows you to write data efficiently with theupsert function. The Java SDK covers core CDF functionality, such as contextualization and data extraction. You’ll find all the info you need in the cdf-sdk-java repo, and we have updated the Cognite API documentation with code samples on how to perform tasks with the Java SDK.Change isStep setting for ingesting time series to CDF
When you stream time series data points into the CDF data model, the interpolation between the data points is controlled by theisStep property on the time series object. You can now change the isStep property value without recreating and re-ingesting the entire time series.February, 2022
Cognite Data Fusion (CDF) streams data into the CDF data model, where the data is normalized and enriched by adding connections between data resources of different types to provide better business decisions. With this release, you can ingest geospatial data into CDF to contextualize and map with other CDF resource types to deliver accurate spatial analysis and statistics.Read more on this and all the other improvements for the CDF February 2022 release below.Geospatial data
Work with geospatial data using the Geospatial API and Python & JavaScript SDKs to handle the positioning and geometry of CDF data. We currently support a multitude of coordinate reference systems (CRS), but you can add other entries from the European Petroleum Survey Group (EPSG) as a custom CRS. You can also transform an input CRS to the defined CRS to improve the search in geometrics.Ingest PI Asset Framework data to CDF
The new Cognite PI Asset Framework (AF) extractor is considered an extension to the Cognite PI extractor. The Cognite PI AF extractor retrieves a dump of the asset hierarchy from the Asset Framework on the PI Server and ingests this into Cognite RAW. Subsequently, the extractor regularly checks for updates in the PI asset framework and ingests any changes into CDF.Delete asset hierarchies using CDF Transformations
You can now delete asset hierarchies in CDF using CDF Transformations. Items are deleted by their internal ID, and non-existing IDs are ignored. Read more about CDF Transformations.Ingest data into CDF Relationships using PostgreSQL gateway
We’re rolling out the option to use the PostgreSQL gateway and Azure Data Factory (ADF) to ingest data directly into CDF Relationships. Select the public.relationships table in your ADF sink.Ingest time series data points directly into CDF Data points using the Cognite DB extractor
The Cognite DB extractor, version 2.4.1, provides a new option to ingest data points directly into CDF Data points instead of Cognite RAW. Set destination type as TIME_SERIES in your configuration file.Early adopter programs
Cognite offers several Early adopter programs to our users. We encourage you to take this opportunity to learn about and test our features currently in the making. Navigate to Cognite Hub, open the Group tab, and select Request to join to become an Early adopter candidate.December, 2021
Trusting the data in Cognite Data Fusion (CDF) is the core of managing, building, and using the Cognite product suite. Data engineers need sufficient insight into the staged data to take the next step on the ingestion journey and the new data profiling report on staged data in Cognite RAW caters for this.This CDF release also introduces the new Cognite Solutions Portal. The Cognite Solutions Portal collects applications, use cases, and other helpful links to solutions your organization has built on top of CDF. Users no longer have to keep track of many different URLs to access the information they need in their daily work.Read more about this and all the other improvements for the CDF December release below.Data profiling on staged data
To get in-depth knowledge about the data quality, discover patterns, outliers, and see other statistics in the new standard profiling report available on the Profile tab in Cognite RAW. Report your findings to the data owners and use this as a trigger to discuss topics such as the best fit for the primary key column and contextualization and to provide the best support for the end-users of the data. Keep iterating on the data integrations to improve the data quality and prepare the data transformation into the CDF data model. Read more here.Get notified about interrupted data flow into CDF
Catch any issues that interrupt the data flow into CDF by setting up email notifications on the Extraction pipeline page. You can define the time condition for triggering the email. Learn how to set it up in this article.New Transformations CLI
The new Transformations CLI replaces the old Jetfire CLI offering a better developer experience. Now, you can easily declare and manage variables in the configuration file without any hard-code. You install the CLI using the Python pip package manager.Read more here.APIs and Python SDK for Transformations
The Transformations APIs and Python SDK are promoted to Version 1. Using the APIs/SDK, data engineers can orchestrate transformations sequentially, making it more reliable and quick. Check out the API docs and SDK docs for more information.Access management for SQL Transformations
You can now manage access to SQL Transformations using the newtransformations:read and transformations:write capabilities in your CDF group.We still support transformation access using the transformations or jetfire groups for existing users to ensure backward compatibility. However, we recommend that you switch to managing access to Transformations using the new capabilities.Introducing the Cognite Solutions Portal
As companies ramp up their digitalization and transformation efforts, the number of applications, dashboards, and use cases increases rapidly. As a result, end-users have to keep track of many different URLs to access the information they need. This is the problem Cognite Solutions Portal sets out to solve. It gathers all your CDF-enabled solutions in one place.Use the Cognite Solutions Portal to add applications, dashboards from analytics and visualization tools, and helpful links into shared spaces.Explore data in 3D
The CDF Data explorer already helps solution builders and data scientists find, validate, and learn about the data in CDF. Get the full experience by navigating and searching for assets and viewing the details in 3D models on the new 3D tab in the CDF Data explorer.Cognite Charts
Cognite Charts runs on top of CDF and is a powerful tool for engineers and domain experts to explore, trend, and analyze industrial data. For more information, visit the Charts Early Adopter Group on the Cognite Hub.This release has the following improvements:Automatic data alignment
Charts now automatically handle data alignment in your calculations. Time series data often have different sampling frequencies and time stamps, and it can be hard to do even the simplest calculations (e.g. timeseries_1 + timeseries_2), especially if you’re not proficient in Python. Calculation results will be inaccurate and untrustworthy without properly accounting for data alignment.A new and improved no-code calculation builder experience
The no-code calculation builder has been redesigned and rebuilt on a new, powerful library.