Configuration settings - Cognite Docs

To configure the EDM extractor, you must create a configuration file. The file must be in YAML format.

You can use the sample minimal configuration file as a starting point for your configuration settings.

You can set parameters through environment variables using the pattern <configuration section> <parameter>. The environment variables override values set in the configuration file (including default values for parameters not listed in the configuration file). For example, setting the COGNITE_PROJECT environment variable overrides the project parameter under the Cognite section (the matching is case insensitive).

Cognite

Include the cognite section to configure which CDF project the extractor loads data into and how to connect to the project. This section is mandatory and should always contain the project and authentication configuration.

Parameter	Description
`clientId`	Insert the CDF client ID. This is mandatory if you’re using OIDC authentication.
`clientSecret`	Insert the CDF client secret for your CDF project. This is mandatory if you’re using OIDC authentication.
`tokenUrl`	Insert the token URL. This is mandatory if you’re using OIDC authentication. You’ll find further details on the OIDC token URL in authorization and groups.
`apiUrl`	Insert the base URL of the CDF project.
`project`	Insert the CDF project name you want to ingest data into. This field is required.
`rawDatabaseName`	Enter the name of the CDF RAW database to extract data into. If it doesn’t already exist, the extractor creates the database.
`queueBufferSize`	Enter the size for the queue between extraction and writing objects to CDF RAW. The default value is 500. A larger queue will allow for buffering more extracted entities from EDM, but lead to higher memory usage.

Logging

Include the logging section to set up logging to a standard output, such as a terminal window.

Parameter	Description
`level`	Select the verbosity level for console logging. Valid options are `trace`, `debug`, `info`, `warn`, and `error`.
`cdfLevel`	Set the log level for the CDF Java SDK. Valid options are `trace`, `debug`, `info`, `warn`, and `error`. This is an optional value. The default value is `warn`.

EDM

Include the edm section to configure the parameters needed to connect to your EDM instance.

Parameter	Description
`dsisAuthServer`	If you set `dsisAuthType` to `token`, enter the URL for DSIS authentication
`dsisAuthType`	Select how to authenticate to DSIS. Valid options are `token` and `basic`. `token` expects the bearer token from the DSIS authentication server. `basic` uses `dsisUsername` and `dsisPassword` to authenticate.
`verifyDsisAuthServerCertificate`	Set to `false` to disable certificate verification for the DSIS auth Server specified with the `dsisAuthServer` option. This field is optional and verification defaults to `true`.
`gracePeriod`	Enter the time in seconds between each scan for changes to incremental entities. The default value is 180. This field is optional.
`crontabComplete`	Set up a cron schedule for starting the extraction of complete entities.
`runOnStartup`	Set to `true` to run a complete extraction on startup bypassing any schedule set in `crontabComplete`. This is typically used during development. The default value is `false`. This field is optional.
`dsisEdmPath`	Insert the path to the EDM server.
`dsisPathCartoServices`	Insert the path to the EDM Carto services to extract Carto entities from EDM, such as CD_CARTO.
`dsisParallelism`	Insert the number of threads to extract data from EDM. Be aware that a high volume load on the EDM server may cause a crash. The default value is 2. This field is optional.
`processingParallelism`	Insert the number of threads to process and upload extracted entities to CDF RAW. The default value is 4. This field is optional.
`startPadding`	Extractor internal parameter. Do not change.
`measurementSystem`	Enter the measurement system defined in EDM/DSIS. If you set a value, the `measurementsystem` will be appended to all queries. The default value is blank. This field is optional.
`connectTimeout`	Enter the maximum time in seconds to wait for a connection to DSIS. The default value is 5. This field is optional.
`readTimeout`	Set the maximum timeout in seconds for read requests to DSIS. This field is optional. If you don’t enter a value or the value is less then 120, the default value is set to 120.
`entitiesCompleteExtraction`	List the EDM entities to extract completely on each run, even if the entities are incremental. This can be used as a workaround if EDM is missing `create_date` and `update_date` on incremental entities.
`oDataConsumerDebug`	Enable debug logging of EDM traffic for troubleshooting. Valid strings are `ALL`, `REQUEST_HEADER`, `REQUEST_FULL`, `RESPONSE_HEADER`, `RESPONSE_FULL`, `OFF`. This field is optional.
`cartoSync`	Set to `true` to extract the EDM cartography database used (entities `carto` and `carto alias`). These will be extracted into CDF RAW tables named `Carto` and `CartoAlias`, respectively.
`dsisBaseUrl`	Insert the base URL of the DSIS. The extractor combines `dsisBaseUrl`, `dsisEdmPath`, `dsisEdmVdb`, and `dsisEdmProject` to connect to your EDM instance. This is a required field.
`dsisEdmVdb`	Enter the name of the EDM virtual database. This is a required field.
`dsisEdmProject`	Enter the name of the EDM project. The project name `ALL_PROJECTS` can be used to query across all projects in EDM. This is a required field.
`dsisAuthRealm`	Enter the name of the EDM authentication REALM. This is a required field.
`dsisAuthClientId`	Enter the DSIS client ID. This is a required field.
`dsisUsername`	Enter a valid DSIS username. This is a required field.
`dsisPassword`	Enter a valid DSIS password. This is a required field.
`entities`	List the EDM entities you want to extract. See also the `entities` section below.
`consistencySchedule`	Set up a cron schedule for triggering a consistency check of the state of RAW compared to EDM to mark deleted objects in EDM as deleted in RAW.
`consistencyOnStartup`	Set to `true` to run a consistency check between the state in RAW and EDM on startup bypassing any schedule set in `consistencySchedule`.

Entities

Include the entities section to list the entities you want to extract from the EDM server to CDF. You’ll find the list of supported EDM entities here.

Parameter	Description
`name`	Enter the name of the EDM entity, for instance, CD_WELL. This is a required field.
`filter`	OData-based custom filters for EDM entity level. Use this to filter unwanted records from EDM.
`includeFields`	List of properties on the entity to include in the extraction. If not specified, all fields are included. This field can’t be combined with `excludefields`.
`excludeFields`	List of properties to exclude from the extraction of an entity. Don’t use this parameter together with `includeFields`.
`overrideIncremental`	This parameter is only relevant when the entity is listed in the `entitiesCompleteExtraction` parameter and confirms for the extractor that it should both extract incrementally and complete. The entry under `entitiesCompleteExtraction` parameter should have a filter specified to only select entities that are missed by the incremental extraction. Use in combination with a filter to get all entities with create/update_date set to NULL. See the sample configuration file. Use this parameter for special cases only. By default, this parameter is disabled.
`batchSize`	The number of entities to retrieve per request. The default value is 8, or 50 for the `related` parameter. Some heavy entities, such as CD_WELL, have further configuration in the extractor to reduce the number of entities to retrieve in a batch to reduce the load on the EDM server. This value will override these built-in standard values. This field is optional.
`related`	Set to `true` to include related entities in the object extracted. By default set to `true`.
`relatedList`	Set to an optional list of related entities to include. If not set, all related entities are included.
`consistencyCheck`	Set to `false` to exclude an entity from the set of entities that are consistency checked periodically (according to the schedule set in `consistencySchedule`). The consistency check verifies whether all extracted objects in RAW still exist in EDM. If they don’t exist, they’re marked as deleted in RAW. The default value is `true`.

​Cognite

​Logging

​EDM

​Entities

Cognite

Logging

EDM

Entities