Configuration settings - Cognite Docs

To configure the WITSML extractor, create a configuration file in YAML format. You can omit optional parameters, and the extractor applies the documented defaults. The example below shows the minimal configuration needed to run the WITSML extractor in simple mode.

showLineNumbers

extractor:
  CONFIG_DATASET_ID: <the data set ID for temporary files created by the extractor>
  DATA_DATASET_ID: <The data set ID for the CDF resources created by the extractor>

cdf:
  COGNITE_PROJECT: <The CDF project name>
  TENANT_ID: <The ID of your tenant>
  TOKEN_CLIENT_ID: <Client ID for the WITSML extractor>
  TOKEN_CLIENT_SECRET: <Client secret for the WITSML extractor>
  CDF_CLUSTER: <The CDF cluster>

extract:
  'your-witsml-server-reference':
    gateway:
      host: <WITSML server host>
      user: <WITSML user name>
      password: <WITSML password>
    rules:
      well:
        schedule: '*/10 * * * *'
        object_type: WELL

Additional configuration examples are available in the WITSML extractor configuration samples repository.

Extraction rules

You can configure extraction rules to define what, how, and when the extractor ingests WITSML data into CDF. Set the rule_type parameter on each rule.

ChangedObject (CHANGEDOBJECT) — The default rule type. Finds objects that changed since the last request to the WITSML server. The rule requires that the WITSML server sets the dTimeLastUpdated flag correctly. The rule has been tested and verified for Petrolink PetroVault and Kongsberg SiteCom.
UpdateStatus (UPDATESTATUS) — Finds objects to ingest based on a status attribute that matches the rule. The query can become out of sync when the status changes and no rule captures the changed attribute. The rule runs its query and compares the result with data in CDF RAW. If there is a mismatch, the extractor creates a ScheduledObjectQuery for the object. The rule can handle the wellbore isActive status or the log objectGrowing flag.

If you add new rules to the configuration file at runtime, the extractor sets all existing rules to inactive before ingesting the new rules into the table based on their definitions.
If the extractor section isn’t added to the configuration file at runtime, the extractor uses the rules stored in the extractionrules table in the witsml-config database.

Parameter interactions for ChangedObject rules

Three parameters control how ChangedObject rules query WITSML servers and use extraction state. Understanding how they interact helps you tune performance and recover from sync issues without manually editing state stores.

Parameter	What it controls	Default
`filter_on_last_modified`	When `true`, the extractor tracks the last modification time in its state store and only requests objects changed since the previous run. When `false`, the extractor requests all matching objects on every run.	`true`
`only_for_active_wellbores`	When `true`, the extractor limits queries to wellbores marked as active in CDF RAW. This reduces query scope for growing objects, such as logs.	`false`
`load_deltas`	For growing objects, when `true`, the extractor processes only new data points since the last run. When `false`, the extractor reprocesses all data points on every run.	`true`

filter_on_last_modified and load_deltas both rely on extraction state, but they apply at different stages. filter_on_last_modified filters which objects the extractor queries from the WITSML server. load_deltas controls whether the extractor re-ingests all data points for a growing object or only new points. only_for_active_wellbores is independent of modification timestamps. It restricts queries to wellbores with an active status in CDF RAW, which is useful when you ingest logs only for active wellbores.

Setting filter_on_last_modified to false triggers a full re-fetch on every scheduled run for that rule. This increases load on the WITSML server and CDF. Use it temporarily for backfill or re-sync, then set it back to true.

Backfill or re-sync data

If extraction state is out of sync with the WITSML server, or you need to re-ingest historical data, temporarily disable incremental filtering instead of manually editing state stores.

Disable incremental filtering

Set filter_on_last_modified to false on the affected rule or rules in your configuration file.

rules:
  wellbore.all:
    schedule: '*/10 * * * *'
    object_type: WELLBORE
    config:
      filter_on_last_modified: false

Run the extractor

Restart the extractor or wait for the next scheduled run. The extractor performs a full re-fetch for the affected rules.

Re-enable incremental filtering

After the backfill completes, set filter_on_last_modified back to true (or remove the parameter to use the default).

Confirm that subsequent runs fetch only changed objects and extraction state updates normally.

If you also need to reprocess all data points for growing objects such as logs, set load_deltas to false during the backfill. Set it back to true after the re-sync completes.

Extractor

Include the extractor section to configure the extractor setup.

Parameter	Description	Required	Default
`APP_NAME`	Name of the extractor deployment. Used in the extraction pipeline run in CDF.	No	`witsml-extractor`
`MODE`	Execution mode for the extractor.	No	`SIMPLE`
`JSON_LOGGING`	Set to `true` to enable debug logging in JSON format. Useful for troubleshooting.	No	`false`
`LOG_LEVEL`	Logging verbosity. Valid options, in decreasing verbosity, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`.	No	`INFO`
`CONFIG_DATASET_ID`	Data set ID for the WITSML configuration in CDF RAW.	No	—
`DATA_DATASET_ID`	Data set ID for the WITSML data in CDF RAW.	No	—
`RAW_DB_FOR_CONFIG`	Name of the CDF RAW database for the WITSML configuration.	No	`witsml-config`
`RAW_DB_FOR_DATA`	Name of the CDF RAW database for the WITSML data.	No	`witsml-data`
`DEPTH_TO_ROWNUMBER_SCALE_FACTOR`	Factor to multiply depth indexes for `row-key` in sequences.	No	`1000`
`EXTPIPELINE_EXT_ID_PREFIX`	Prefix added to the extraction pipeline run in CDF.	No	—
`EXT_ID_PREFIX`	External ID prefix for objects created directly in the CDF resource type.	No	—
`EXT_ID_SUFFIX`	External ID suffix for objects created directly in the CDF resource type.	No	—
`EXT_ID_MNEMONIC_LOWERCASE`	Set to `true` to convert log curve mnemonics to lowercase in external IDs.	No	`true`
`FIND_UNAVAILABLE_IN_SOURCE`	Set to `true` for maintenance jobs that look for objects in CDF that are no longer available in the source system.	No	`true`
`ADD_LOG_INFO_TO_TIMESERIES`	Set to `true` to add log header information to time series created by the extractor.	No	`true`
`ARCHIVE_DOWNLOADED_FILES`	Set to `true` to archive and compress complete XML files downloaded from the WITSML server. Use when you reprocess the same file in different environments or test different configurations.	No	`false`
`DELETE_EMPTY_LOGS`	Set to `true` to delete empty log objects from CDF after processing.	No	`false`
`ARCHIVE_RESPONSE_FILES`	Set to `true` to keep response XML files downloaded from the WITSML server after processing.	No	`true`

CDF

Include the cdf section to configure which CDF project the extractor loads data into and how to connect to the project. This section is required.

Parameter	Description	Required	Default
`COGNITE_PROJECT`	CDF project name to ingest data into.	Yes	—
`TENANT_ID`	Azure tenant ID.	Yes	—
`TOKEN_CLIENT_ID`	CDF client ID for OIDC authentication.	Yes	—
`TOKEN_CLIENT_SECRET`	CDF client secret for OIDC authentication.	Yes	—
`CDF_CLUSTER`	Name of the CDF cluster.	Yes	—

ETP

Include the etp section when you set up ingestion of live data from WITSML ETP objects to CDF. This section is optional.

Parameter	Description	Required	Default
`keep_etp_msg`	Keep the ETP message stored in the queue after processing.	No	`false`
`trigger_refresh_lag`	Trigger a WebSocket reconnection when latency exceeds 10 minutes.	No	`false`
`refresh_after`	WebSocket session refresh interval in seconds. The ETP client closes and reopens the session after this duration.	No	`300`
`max_msg_rate`	Maximum ETP message rate while streaming ETP records. Defined by Energistics.	No	`100`
`max_data_items`	Maximum number of data items per ETP message. Defined by Energistics.	No	`1100`
`index_padding_minutes`	Index padding time in minutes when the ETP client starts WebSocket communication.	No	`10`
`filter_uids`	Filter WITSML logs to listen to by log UID. An empty list fetches all active logs.	No	`[]`
`max_queue_size`	Maximum upload queue size. Upload to CDF triggers when this limit is reached.	No	`1000`
`message_timeout`	ETP WebSocket communication timeout in seconds.	No	`30`

ETP gateway

Include the gateway subsection to configure how the extractor connects to the WITSML ETP provider.

Parameter	Description	Required	Default
`host`	Base URL of the WITSML server.	Yes	—
`user`	Username for authenticating to the WITSML server.	Yes	—
`password`	Password for authenticating to the WITSML server.	Yes	—

Extract

This section contains the parameters needed to connect to your WITSML server and the related extraction rules. You can configure several WITSML servers. Each server needs its own witsml-server-reference with gateway and rules sections. The server reference is stored on all main object rows in CDF RAW to reference the object source.

Gateway

Include the gateway subsection to configure how the extractor connects to the WITSML server.

Parameter	Description	Required	Default
`host`	Base URL of the WITSML server.	Yes	—
`user`	Username for authenticating to the WITSML server.	Yes	—
`password`	Password for authenticating to the WITSML server.	Yes	—

Rules

Include the rules subsection to define what, how, and when to ingest WITSML data into CDF. See Extraction rules for more details.

Parameter	Description	Required	Default
`schedule`	Schedule for the rule. Use cron expressions enclosed in `""` or interval syntax such as `s:10`.	Yes	—
`object_type`	WITSML data object type that the rule extracts and ingests into CDF. Enter this value in uppercase.	Yes	—
`rule_type`	Rule type. Valid options are `CHANGEDOBJECT` and `UPDATESTATUS`.	No	`CHANGEDOBJECT`
`base_query`	Custom WITSML query for the object type. If omitted, the extractor uses the standard query for the given object type.	No	Standard query for `object_type`
`config`	Rule-specific settings. Required for `UPDATESTATUS` rules. Optional for `CHANGEDOBJECT` rules; the extractor applies defaults when omitted. See the sections below.	Conditional	See rule type sections

The following values are valid for object_type.

ATTACHMENT
BHARUN
CEMENTJOB
DRILLREPORT
FLUIDSREPORT
FORMATIONMARKER
LOG
DEPTHLOG
TIMELOG
MESSAGE
MUDLOG
RIG
RISK
TRAJECTORY
WELL
TUBULAR
WBGEOMETRY
WELLBORE

ChangedObject rule config

These parameters apply when rule_type is CHANGEDOBJECT (or omitted). You can omit the entire config section to use all defaults.

Parameter	Description	Required	Default
`load_deltas`	Set to `true` to process only new items for growing objects. Set to `false` to process all items every time.	No	`true`
`ingest_to_clean`	Set to `true` to create objects directly in the CDF resource type. Applies only to attachment and log objects.	No	`false`
`only_for_active_wellbores`	Set to `true` to limit queries to active wellbores in CDF RAW. Improves performance when the extractor looks for changes in growing objects.	No	`false`
`filter_on_last_modified`	Set to `true` to query only objects changed since the last run. Set to `false` to retrieve all matching objects on every run. See Backfill or re-sync data.	No	`true`
`log_data_to_raw_variant`	How to store log data records in CDF RAW. Valid options are `NONE`, `ALL`, `ONLY_TIME`, and `ONLY_DEPTH`. Applies only to log object types.	No	`NONE`

UpdateStatus rule config

These parameters apply when rule_type is UPDATESTATUS. The config section and all parameters below are required.

Parameter	Description	Required	Default
`attribute`	Attribute name in the WITSML object to compare. Typically `isActive` for wells and `objectGrowing` for logs.	Yes	—
`look_for_value`	Attribute value to compare with CDF RAW to find out-of-sync objects. Typically `true` when comparing `isActive`.	Yes	—
`ingest_rule_ref`	Name of the rule to run when the extractor finds out-of-sync objects.	Yes	—
`only_for_active_wellbores`	Set to `true` to limit queries to active wellbores in CDF RAW. Improves performance when looking for changes in growing objects.	No	`false`

​Extraction rules

​Parameter interactions for ChangedObject rules

​Backfill or re-sync data

​Extractor

​CDF

​ETP

​ETP gateway

​Extract

​Gateway

​Rules

​ChangedObject rule config

​UpdateStatus rule config

Extraction rules

Parameter interactions for ChangedObject rules

Backfill or re-sync data

Extractor

CDF

ETP

ETP gateway

Extract

Gateway

Rules

ChangedObject rule config

UpdateStatus rule config