This article documents the ongoing improvements we're making to the Cognite PI Extractor.
# Version 2.1 - September 03, 2020
The PI Extractor - version 2.1 contains a number of improvements. Here are the main changes:
# New features
- Stream real-time data from PI using the Observer pattern. The data points are fetched from PI with a higher frequency and added to a queue on the extractor side. This reduces the number of data loss incidents caused by queue overflow on the server.
- PI Replace Utility - version 2 with improved functionality that can read the data loss/reconnection events created by the PI Extractor.
- The extractor can search CDF time series by dataset ID and delete the time series that are available in CDF but not in PI.
- The Extractor Utils Library is a collection of utilities for building extractors. Using this library makes the PI Extractor codebase smaller and easier to maintain. Duplicated code among extractors is removed and the library’s better test coverage indicates improved code quality.
- Data loss and reconnection incidents are saved in CDF as Events. These events can be used for quality checks.
- The time ranges of extracted time series were sometimes incorrectly expanded when streamer and backfiller were running simultaneously, i.e., the historical events from the streaming could affect the range of time series that were still backfilling.
- Deleted PI Points no longer cause the PI Extractor to fail and reconnect to the PI Server.