> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognite.com/llms.txt > Use this file to discover all available pages before exploring further. # Configuration settings > Configure the SAP extractor with YAML files to connect to SAP instances and extract data to CDF. To configure the SAP extractor, you must create a configuration file. The file must be in [YAML](https://yaml.org) format. You can use the sample minimal configuration file included with the extractor packages as a starting point for your configuration settings. The configuration file contains the global parameter `version`, which holds the version of the configuration schema. This article describes version 1. You can set up [extraction pipelines](/cdf/integration/guides/interfaces/configure_integrations) to use versioned extractor configuration files stored in the cloud. ## Using values from environment variables The configuration file allows substitutions with environment variables. For example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} cognite: secret: ${COGNITE_CLIENT_SECRET} ``` will load the value from the `COGNITE_CLIENT_SECRET` environment variable into the `cognite/secret` parameter. You can also do string interpolation with environment variables, for example: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN} ``` Implicit substitutions only work for unquoted value strings. For quoted strings, use the `!env` tag to activate environment substitution: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} url: !env 'http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}' ``` ## Using values from Azure Key Vault The DB extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the `!keyvault` tag followed by the name of the secret you want to load. For example, to load the value of the `my-secret-name` secret in Key Vault into a `password` parameter, configure your extractor like this: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} password: !keyvault my-secret-name ``` To use Key Vault, you also need to include the `azure-keyvault` section in your configuration, with the following parameters: | Parameter | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `keyvault-name` | Name of Key Vault to load secrets from | | `authentication-method` | How to authenticate to Azure. Either `default` or `client-secret`. For `default`, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). For `client-secret`, the extractor will authenticate with a configured client ID/secret pair. | | `client-id` | Required for using the `client-secret` authentication method. The client ID to use when authenticating to Azure. | | `secret` | Required for using the `client-secret` authentication method. The client secret to use when authenticating to Azure. | | `tenant-id` | Required for using the `client-secret` authentication method. The tenant ID of the Key Vault in Azure. | **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} azure-keyvault: keyvault-name: my-keyvault-name authentication-method: client-secret tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648 client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6 secret: 1234abcd ``` Base configuration object | Parameter | Type | Description | | ------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | either string or integer | Configuration file version | | `type` | either `local` or `remote` | Configuration file type. Either `local`, meaning the full config is loaded from this file, or `remote`, which means that only the `cognite` section is loaded from this file, and the rest is loaded from extraction pipelines. Default value is `local`. | | [`cognite`](#cognite) | object | The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | | [`logger`](#logger) | object | The optional `logger` section sets up logging to a console and files. | | [`metrics`](#metrics) | object | The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | | [`sap`](#sap) | list | List of SAP instances to connect to | | [`endpoints`](#endpoints) | list | List of endpoints to query | | [`extractor`](#extractor) | object | General extractor configuration | ## `cognite` Global parameter. The cognite section describes which CDF project the extractor will load data into and how to connect to the project. | Parameter | Type | Description | | ----------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `project` | string | Insert the CDF project name. | | [`idp-authentication`](#cognite.idp-authentication) | object | The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | | [`data-set`](#cognite.data-set) | object | Enter a data set the extractor should write data into | | [`extraction-pipeline`](#cognite.extraction-pipeline) | object | Enter the extraction pipeline used for remote config and reporting statuses | | `host` | string | Insert the base URL of the CDF project. Default value is `https://api.cognitedata.com`. | | `timeout` | integer | Enter the timeout on requests to CDF, in seconds. Default value is `30`. | | `external-id-prefix` | string | Prefix on external ID used when creating CDF resources | | [`connection`](#cognite.connection) | object | Configure network connection details | ### `idp-authentication` Part of [`cognite`](#cognite) configuration. The `idp-authentication` section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory). | Parameter | Type | Description | | -------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `authority` | string | Insert the authority together with `tenant` to authenticate against Azure tenants. Default value is `https://login.microsoftonline.com/`. | | `client-id` | string | **Required.** Enter the service principal client id from the IdP. | | `tenant` | string | Enter the Azure tenant. | | `token-url` | string | Insert the URL to fetch tokens from. | | `secret` | string | Enter the service principal client secret from the IdP. | | `resource` | string | Resource parameter passed along with token requests. | | `audience` | string | Audience parameter passed along with token requests. | | [`scopes`](#cognite.idp-authentication.scopes) | list | Enter a list of scopes requested for the token | | `min-ttl` | integer | Insert the minimum time in seconds a token will be valid. If the cached token expires in less than `min-ttl` seconds, it will be refreshed even if it is still valid. Default value is `30`. | | [`certificate`](#cognite.idp-authentication.certificate) | object | Authenticate with a client certificate | #### `scopes` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Enter a list of scopes requested for the token Each element of this list should be a string. #### `certificate` Part of [`idp-authentication`](#cognite.idp-authentication) configuration. Authenticate with a client certificate | Parameter | Type | Description | | --------------- | ------ | ------------------------------------------------------------------------------------------ | | `authority-url` | string | Authentication authority URL | | `path` | string | **Required.** Enter the path to the .pem or .pfx certificate to be used for authentication | | `password` | string | Enter the password for the key file, if it is encrypted. | ### `data-set` Part of [`cognite`](#cognite) configuration. Enter a data set the extractor should write data into | Parameter | Type | Description | | ------------- | ------- | -------------------- | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `extraction-pipeline` Part of [`cognite`](#cognite) configuration. Enter the extraction pipeline used for remote config and reporting statuses | Parameter | Type | Description | | ------------- | ------- | -------------------- | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `connection` Part of [`cognite`](#cognite) configuration. Configure network connection details | Parameter | Type | Description | | ---------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `disable-gzip` | boolean | Whether or not to disable gzipping of json bodies. | | `status-forcelist` | string | HTTP status codes to retry. Defaults to 429, 502, 503 and 504 | | `max-retries` | integer | Max number of retries on a given http request. Default value is `10`. | | `max-retries-connect` | integer | Max number of retries on connection errors. Default value is `3`. | | `max-retry-backoff` | integer | Retry strategy employs exponential backoff. This parameter sets a max on the amount of backoff after any request failure. Default value is `30`. | | `max-connection-pool-size` | integer | The maximum number of connections which will be kept in the SDKs connection pool. Default value is `50`. | | `disable-ssl` | boolean | Whether or not to disable SSL verification. | | [`proxies`](#cognite.connection.proxies) | object | Dictionary mapping from protocol to url. | #### `proxies` Part of [`connection`](#cognite.connection) configuration. Dictionary mapping from protocol to url. ## `logger` Global parameter. The optional `logger` section sets up logging to a console and files. | Parameter | Type | Description | | ---------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- | | [`console`](#logger.console) | object | Include the console section to enable logging to a standard output, such as a terminal window. | | [`file`](#logger.file) | object | Include the file section to enable logging to a file. The files are rotated daily. | | `metrics` | boolean | Enables metrics on the number of log messages recorded per logger and level. This requires `metrics` to be configured as well | ### `console` Part of [`logger`](#logger) configuration. Include the console section to enable logging to a standard output, such as a terminal window. | Parameter | Type | Description | | --------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for console logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | ### `file` Part of [`logger`](#logger) configuration. Include the file section to enable logging to a file. The files are rotated daily. | Parameter | Type | Description | | ----------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `level` | either `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` | Select the verbosity level for file logging. Valid options, in decreasing verbosity levels, are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. Default value is `INFO`. | | `path` | string | **Required.** Insert the path to the log file. | | `retention` | integer | Specify the number of days to keep logs for. Default value is `7`. | ## `metrics` Global parameter. The `metrics` section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a [Prometheus pushgateway](https://prometheus.io), but you can also send metrics as time series in the CDF project. | Parameter | Type | Description | | ----------------------------------------- | ------ | ------------------------------------------------------------------------------------------------- | | [`push-gateways`](#metrics.push-gateways) | list | List of prometheus pushgateway configurations | | [`cognite`](#metrics.cognite) | object | Push metrics to CDF timeseries. Requires CDF credentials to be configured | | [`server`](#metrics.server) | object | The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | ### `push-gateways` Part of [`metrics`](#metrics) configuration. List of prometheus pushgateway configurations Each element of this list should be a the push-gateways sections contain a list of metric destinations. | Parameter | Type | Description | | --------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `host` | string | Enter the address of the host to push metrics to. | | `job-name` | string | Enter the value of the `exported_job` label to associate metrics with. This separates several deployments on a single pushgateway, and should be unique. | | `username` | string | Enter the credentials for the pushgateway. | | `password` | string | Enter the credentials for the pushgateway. | | `clear-after` | either null or integer | Enter the number of seconds to wait before clearing the pushgateway. When this parameter is present, the extractor will stall after the run is complete before deleting all metrics from the pushgateway. The recommended value is at least twice that of the scrape interval on the pushgateway. This is to ensure that the last metrics are gathered before the deletion. Default is disabled. | | `push-interval` | integer | Enter the interval in seconds between each push. Default value is `30`. | ### `cognite` Part of [`metrics`](#metrics) configuration. Push metrics to CDF timeseries. Requires CDF credentials to be configured | Parameter | Type | Description | | --------------------------------------- | ------- | ------------------------------------------------------------------------------------------------ | | `external-id-prefix` | string | **Required.** Prefix on external ID used when creating CDF time series to store metrics. | | `asset-name` | string | Enter the name for a CDF asset that will have all the metrics time series attached to it. | | `asset-external-id` | string | Enter the external ID for a CDF asset that will have all the metrics time series attached to it. | | `push-interval` | integer | Enter the interval in seconds between each push to CDF. Default value is `30`. | | [`data-set`](#metrics.cognite.data-set) | object | Data set the metrics will be created under | #### `data-set` Part of [`cognite`](#metrics.cognite) configuration. Data set the metrics will be created under | Parameter | Type | Description | | ------------- | ------- | -------------------- | | `id` | integer | Resource internal id | | `external-id` | string | Resource external id | ### `server` Part of [`metrics`](#metrics) configuration. The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping | Parameter | Type | Description | | --------- | ------- | ----------------------------------------------------------------------- | | `host` | string | Host to run the prometheus server on. Default value is `0.0.0.0`. | | `port` | integer | Local port to expose the prometheus server on. Default value is `9000`. | ## `sap` Global parameter. List of SAP instances to connect to Each element of this list should be a configuration of an SAP source Either one of the following options: * [SAP NetWeaver gateway](#sap.sap_netweaver_gateway) * [SAP SOAP Source](#sap.sap_soap_source) ### `sap_netweaver_gateway` Part of [`sap`](#sap) configuration. The SAP NetWeaver Gateway lets clients connect using the Open Data Protocol (OData). **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: odata source-name: mys4hana gateway-url: https://mys4hana.com/sap/opu/odata/sap/ client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: odata4 source-name: mys4hana gateway-url: https://mys4hana.com/sap/opu/odata4/sap client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` | Parameter | Type | Description | | --------------------------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | always `odata` | **Required.** Type of SAP source connection, set to `odata` for SAP OData v2 sources. Set to `odata4` for SAP OData v4 sources. | | `source-name` | string | **Required.** Enter a name for the source that will be used throughout the `endpoints` section and for logging. The name must be unique for each source in the configuration file. | | `gateway-url` | string | **Required.** Insert the SAP NetWeaver Gateway URL | | `client` | string | **Required.** Insert the SAP client number | | `username` | string | **Required.** Enter the SAP username to connect to the SAP NetWeaver Gateway. | | `password` | string | **Required.** Enter the SAP password to connect to the SAP NetWeaver Gateway | | `disable-ssl` | boolean | Disable the SSL certificate verification towards the SAP destination. Default value is set to False. | | `connection_check_timeout` | integer | Enter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is `60`. | | `language` | string | Enter the sap-language URL parameter. The default value is EN | | [`certificates`](#sap.sap_netweaver_gateway.certificates) | object | Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | | [`proxy`](#sap.sap_netweaver_gateway.proxy) | object | HTTP and/or HTTPS proxies to be used when connecting to the SAP source system. | | `timezone` | either `local` or `utc` | Specify how the extractor should handle the source time zones. Default value is `local`. | #### `certificates` Part of [`sap_netweaver_gateway`](#sap.sap_netweaver_gateway) configuration. Certificates needed for authentication towards SAP instance. There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | Parameter | Type | Description | | ------------- | ------ | -------------------------------------------------------- | | `ca-cert` | string | **Required.** Enter the path to the CA certificate file. | | `public-key` | string | **Required.** Enter the path to the public key file. | | `private-key` | string | **Required.** Enter the path to the private key file. | #### `proxy` Part of [`sap_netweaver_gateway`](#sap.sap_netweaver_gateway) configuration. HTTP and/or HTTPS proxies to be used when connecting to the SAP source system. | Parameter | Type | Description | | --------- | ------ | ------------------------------------- | | `http` | string | Enter the address of the HTTP proxy | | `https` | string | Enter the address of the HTTPS proxy. | ### `sap_soap_source` Part of [`sap`](#sap) configuration. The SAP extractor can connect to SAP instances using SOAP, such as for SAP ERP SOAMANAGER **Example:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} type: soap source-name: soap-funcloc wsdl-url: https://myerp.com/sap/bc/srt/wsdl/sap/bc/srt/rfc/sap/test/100/test_funcloc/test_funcloc?sap-client=100 client: '100' username: ${SAP_USERNAME} password: ${SAP_PASSWORD} language: EN ``` | Parameter | Type | Description | | --------------------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | always `soap` | **Required.** Type of SAP source connection, set to `soap` for SAP SOAP sources. | | `source-name` | string | **Required.** Enter a name for the source that will be used throughout the `endpoints` section and for logging. The name must be unique for each source in the configuration file. | | `wsdl-url` | string | **Required.** Insert the [SOAP WSDL URL](https://help.sap.com/doc/saphelp_nw75/7.5.5/en-US/48/d185ca45885c97e10000000a42189d/content.htm?no_cache=true) related to the SAP ABAP webservice | | `strict-parser` | boolean | Flag to control how the extractor behave when parsing SOAP responses. Default value is set to True. | | `disable-ssl` | boolean | Disable the SSL certificate verification towards the SAP destination. Default value is set to False. | | `client` | string | **Required.** Insert the SAP client number | | `username` | string | **Required.** Enter the username to connect to the SAP Webservice. | | `password` | string | **Required.** Enter the password to connect to the SAP Webservice | | `language` | string | Enter the sap-language URL parameter. The default value is EN | | [`certificates`](#sap.sap_soap_source.certificates) | object | Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | | `connection_check_timeout` | integer | Enter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is `60`. | | `session_timeout` | integer | Enter the timeout (in seconds) period the extractor should use when fetching WSDL definitions from the SOAP source. Default value is `300`. | | `timezone` | either `local` or `utc` | Specify how the extractor should handle the source time zones. Default value is `local`. | #### `certificates` Part of [`sap_soap_source`](#sap.sap_soap_source) configuration. Certificates needed for authentication towards SAP instance. There are three certificates needed to perform the authentication: certificate authority (`ca-cert`), public key (`public-key`), and private key (`private-key`). | Parameter | Type | Description | | ------------- | ------ | -------------------------------------------------------- | | `ca-cert` | string | **Required.** Enter the path to the CA certificate file. | | `public-key` | string | **Required.** Enter the path to the public key file. | | `private-key` | string | **Required.** Enter the path to the private key file. | ## `endpoints` Global parameter. List of endpoints to query Each element of this list should be a description of an endpoint to extract data from on one of the configured SAP sources. | Parameter | Type | Description | | ------------------------------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | **Required.** Enter a name of this SAP endpoint that will be used to for logging. The name must be unique for each query in the configuration file. | | `source-name` | string | **Required.** Enter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the [`sap`](#sap) section. | | `sap-service` | string | **Required.** Enter the name of the related SAP service. For `odata` endpoints, it's the SAP OData service. For `soap` endpoints, it's the service defined in the WSDL document. | | `sap-entity` | string | **Required.** Enter the name of the related SAP entity. For `odata` endpoints, it's the name of the OData entity. For `soap` endpoints, it's the name of the SOAP operator as defined in the WSDL | | [`destination`](#endpoints.destination) | configuration for either RAW, Events, Assets, Time Series or Files | **Required.** The destination of the data in CDF. | | `pagination-type` | either `client`, `server` or `no-pagination` | OData pagination type when running full load (non incremental) from SAP. Default value is `no-pagination`. | | [`sap-key`](#endpoints.sap-key) | list | Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. | | [`select`](#endpoints.select) | list | Enter a list of fields to be selected from the SAP entity, using OData \$select operation. This parameter is available only for SAP OData endpoints. | | [`related-entities`](#endpoints.related-entities) | object | Sub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources. | | [`attachments`](#endpoints.attachments) | object | Extraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations) | | `request` | string | Enter the request to be sent to the SAP. This is a required parameter for `soap` endpoints. See the [Requests](#requests) section for more details. | | `incremental-field` | string | Enter the name of the field to be used as reference for the incremental runs. If this field is left out, the extractor will fetch full data loads every run. | | `incremental-field-format` | string | Enter the date format of the incremental field so the extractor can convert it properly when running incremental queries. Example: `%H:%M:%S` is the date format for the date value `12:00:00`. This is an optional parameter. | | `initial-start` | string | Enter the initial value to be used when running incremental requests. Mandatory parameter when running incremental loads in SOAP endpoints. | | [`schedule`](#endpoints.schedule) | configuration for either Fixed interval or CRON expression | Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise.

**Examples:**
`{'schedule': {'type': 'interval', 'expression': '1h'}}`
`{'schedule': {'type': 'cron', 'expression': '0 7-17 * * 1-5'}}` | | [`extract-schema`](#endpoints.extract-schema) | object | If included, the extractor will extract the SAP entity schema to CDF RAW | | `filter` | string | Enter the filter query string. The `$filter` system query option allows clients to filter a collection of resources from the target SAP OData endpoint. This is only relevant for `odata` sources. | | `soap-operation` | string | Enter the name of the SOAP operation defined in the WSDL configuration. For `soap` endpoints, this is a required parameter. | | `soap-port` | string | Enter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for `soap` endpoints. | ### Requests The `request` parameter is part of the [`endpoints`](#endpoints) configuration, and is required `soap` endpoints. #### SOAP requests SAP ABAP Webservices are SOAP/based, meaning the requests to the SAP server must be in a valid XML format. The SAP extractor expects this XML to be added as a string in the `request` parameter. This is an example of a valid XML request to a SAP ABAP Webservice generated from a SAP Function Module: ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} request: | String 57 String 58 S String 60 Strin S String 63 Stri 1010 Str String 67 I 1010 1010 ``` ### `destination` Part of [`endpoints`](#endpoints) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.destination.raw) * [Events](#endpoints.destination.events) * [Assets](#endpoints.destination.assets) * [Time Series](#endpoints.destination.time_series) * [Files](#endpoints.destination.files) #### `raw` Part of [`destination`](#endpoints.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | ---------- | ------------ | ----------------------------------------------------------------------------------------- | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | #### `events` Part of [`destination`](#endpoints.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | -------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ##### `field-mapping` Part of [`events`](#endpoints.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | ------------- | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | #### `assets` Part of [`destination`](#endpoints.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | -------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ##### `field-mapping` Part of [`assets`](#endpoints.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | ------------------ | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | #### `time_series` Part of [`destination`](#endpoints.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | ------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------- | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ##### `field-mapping` Part of [`time_series`](#endpoints.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | ------------ | ------ | ----------------------------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | #### `files` Part of [`destination`](#endpoints.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | --------- | -------------- | -------------------------------------------------------------------------------------- | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | ### `sap-key` Part of [`endpoints`](#endpoints) configuration. Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. Each element of this list should be a string. ### `select` Part of [`endpoints`](#endpoints) configuration. Enter a list of fields to be selected from the SAP entity, using OData \$select operation. This parameter is available only for SAP OData endpoints. Each element of this list should be a string. ### `related-entities` Part of [`endpoints`](#endpoints) configuration. Sub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources. | Parameter | Type | Description | | -------------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `to_expand` | string | Name of the value to be used in the OData \$expand operation | | `sap-service` | string | Enter the name of the related SAP service. For `odata` endpoints, it's the SAP OData service. For `soap` endpoints, it's the SOAP service defined in the WSDL document | | `sap-entity` | string | Enter the name of the related SAP entity. For `odata` endpoints, it's the name of the OData entity. For `soap` endpoints, it's the name of the SOAP object as defined in the WSDL. | | `source-name` | string | Enter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the [`sap`](#sap) section. | | [`destination`](#endpoints.related-entities.destination) | configuration for either RAW, Events, Assets, Time Series or Files | The destination of the data in CDF. | | [`sap-key`](#endpoints.related-entities.sap-key) | list | Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. | | `request` | string | Enter the request to be sent to the SAP when using SOAP as a SAP source. This is a required parameter for `soap` endpoints. See the [Requests](#requests) section for more details. | | `parent-field` | string | Enter the name of the field from the parent entity to be used as input key when querying the related entity. This parameter is mandatory when using SOAP sources. | | `soap-operation` | string | Enter the name of the SOAP operation defined in the WSDL configuration. For `soap` endpoints, this is a required parameter. | | `soap-port` | string | Enter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for `soap` endpoints. | #### `destination` Part of [`related-entities`](#endpoints.related-entities) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.related-entities.destination.raw) * [Events](#endpoints.related-entities.destination.events) * [Assets](#endpoints.related-entities.destination.assets) * [Time Series](#endpoints.related-entities.destination.time_series) * [Files](#endpoints.related-entities.destination.files) ##### `raw` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | ---------- | ------------ | ----------------------------------------------------------------------------------------- | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ##### `events` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | ------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.related-entities.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ###### `field-mapping` Part of [`events`](#endpoints.related-entities.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | ------------- | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | ##### `assets` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | ------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.related-entities.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ###### `field-mapping` Part of [`assets`](#endpoints.related-entities.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | ------------------ | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | ##### `time_series` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | ------------------------------------------------------------------------------------ | -------------------- | ----------------------------------------------------------------------------------------- | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.related-entities.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ###### `field-mapping` Part of [`time_series`](#endpoints.related-entities.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | ------------ | ------ | ----------------------------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | ##### `files` Part of [`destination`](#endpoints.related-entities.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | --------- | -------------- | -------------------------------------------------------------------------------------- | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | #### `sap-key` Part of [`related-entities`](#endpoints.related-entities) configuration. Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using `raw` as the CDF destination. Each element of this list should be a string. ### `attachments` Part of [`endpoints`](#endpoints) configuration. Extraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations) | Parameter | Type | Description | | --------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`destination`](#endpoints.attachments.destination) | configuration for either RAW, Events, Assets, Time Series or Files | The destination of the data in CDF. | | `object-key` | string | SAP primary key of the entity related to the attachment. This is a mandatory parameter. | | `business-object-type` | string | SAP Business Object Type related to the entity related to the attachment. For example, `BUS2038` is the standard business object type for SAP Maintenance Notifications. This is a mandatory parameter. | #### `destination` Part of [`attachments`](#endpoints.attachments) configuration. The destination of the data in CDF. Either one of the following options: * [RAW](#endpoints.attachments.destination.raw) * [Events](#endpoints.attachments.destination.events) * [Assets](#endpoints.attachments.destination.assets) * [Time Series](#endpoints.attachments.destination.time_series) * [Files](#endpoints.attachments.destination.files) ##### `raw` Part of [`destination`](#endpoints.attachments.destination) configuration. The `raw` destination writes data to the CDF staging area (RAW). The `raw` destination requires the `sap-key` parameter in the endpoint configuration. | Parameter | Type | Description | | ---------- | ------------ | ----------------------------------------------------------------------------------------- | | `type` | always `raw` | Type of CDF destination, set to `raw` to write data to RAW. | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ##### `events` Part of [`destination`](#endpoints.attachments.destination) configuration. The `events` destination inserts the resulting data as CDF events. | Parameter | Type | Description | | -------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `events` | Type of CDF destination, set to `events` to write data to CDF Events. | | [`field-mapping`](#endpoints.attachments.destination.events.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Event | ###### `field-mapping` Part of [`events`](#endpoints.attachments.destination.events) configuration. Enter the fields of the SAP entity that corresponds to fields of an Event | Parameter | Type | Description | | ------------- | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the event | | `startTime` | string | Which field in the SAP entity containing the start time of the event | | `endTime` | string | Which field in the SAP entity containing the end time of the event | | `description` | string | Which field in the SAP entity containing the description of the event | | `source` | string | Which field in the SAP entity containing the source of the event | | `type` | string | Which field in the SAP entity containing the type of the event | | `subType` | string | Which field in the SAP entity containing the subtype of the event | ##### `assets` Part of [`destination`](#endpoints.attachments.destination) configuration. The `assets` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | -------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------- | | `type` | always `assets` | Type of CDF destination, set to `assets` to write data to CDF Assets. | | [`field-mapping`](#endpoints.attachments.destination.assets.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of an Asset | ###### `field-mapping` Part of [`assets`](#endpoints.attachments.destination.assets) configuration. Enter the fields of the SAP entity that corresponds to fields of an Asset | Parameter | Type | Description | | ------------------ | ------ | ----------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the asset | | `name` | string | Which field in the SAP entity containing the name of the asset | | `parentExternalId` | string | Which field in the SAP entity containing the perent external ID of the asset | | `description` | string | Which field in the SAP entity containing the description of the asset | | `source` | string | Which field in the SAP entity containing the source of the asset | ##### `time_series` Part of [`destination`](#endpoints.attachments.destination) configuration. The `time_series` destination inserts the resulting data as CDF assets. | Parameter | Type | Description | | ------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------- | | `type` | always `time_series` | Type of CDF destination, set to `time_series` to write data to CDF Time Series. | | [`field-mapping`](#endpoints.attachments.destination.time_series.field-mapping) | object | Enter the fields of the SAP entity that corresponds to fields of a time series data point | ###### `field-mapping` Part of [`time_series`](#endpoints.attachments.destination.time_series) configuration. Enter the fields of the SAP entity that corresponds to fields of a time series data point | Parameter | Type | Description | | ------------ | ------ | ----------------------------------------------------------------------------------------------------- | | `externalId` | string | **Required.** Which field in the SAP entity containing the external ID of the time series to write to | | `timestamp` | string | Which field in the SAP entity containing the timestamp of the datapoint | | `value` | string | Which field in the SAP entity containing the value of the datapoint | ##### `files` Part of [`destination`](#endpoints.attachments.destination) configuration. The `files` destination writes data to the CDF staging area (Files) | Parameter | Type | Description | | --------- | -------------- | -------------------------------------------------------------------------------------- | | `type` | always `files` | Type of CDF destination, set to `files` to write data to RAW. | | `source` | string | Sets the 'Source' metadata field for the related files. This is an optional parameter. | ### `schedule` Part of [`endpoints`](#endpoints) configuration. Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise. Either one of the following options: * [Fixed interval](#endpoints.schedule.fixed_interval) * [CRON expression](#endpoints.schedule.cron_expression) **Examples:** ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: interval expression: 1h ``` ```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}} schedule: type: cron expression: 0 7-17 * * 1-5 ``` #### `fixed_interval` Part of [`schedule`](#endpoints.schedule) configuration. | Parameter | Type | Description | | ------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | always `interval` | **Required.** Type of time interval configuration. Use `interval` to configure a fixed interval. | | `expression` | string | **Required.** Enter a time interval, with a unit. Avaiable units are `s` (seconds), `m` (minutes), `h` (hours) and `d` (days).

**Examples:**
`45s`
`15m`
`2h` | #### `cron_expression` Part of [`schedule`](#endpoints.schedule) configuration. | Parameter | Type | Description | | ------------ | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | always `cron` | **Required.** Type of time interval configuration. Use `cron` to configure CRON schedule. | | `expression` | string | **Required.** Enter a CRON expression. See [crontab.guru](https://crontab.guru) for a guide on writing CRON expressions.

**Example:**
`*/15 8-16 * * *` | ### `extract-schema` Part of [`endpoints`](#endpoints) configuration. If included, the extractor will extract the SAP entity schema to CDF RAW | Parameter | Type | Description | | ---------- | ------------ | ----------------------------------------------------------------------------------------- | | `type` | always `raw` | Type of CDF destination, always set to `raw` | | `database` | string | Enter the CDF RAW database to upload data into. This will be created if it doesn't exist. | | `table` | string | Enter the CDF RAW table to upload data into. This will be created if it doesn't exist. | ## `extractor` Global parameter. General extractor configuration | Parameter | Type | Description | | --------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mode` | either `continuous` or `single` | Extractor mode. Continuous runs the configured queries using the schedules configured per query. Single runs the queries once each. Default value is `single`. | | `parallelism` | integer | Maximum number of parallel queries. Default value is `10`. | | `upload-queue-size` | integer | Maximum size of upload queue. Upload to CDF will be triggered once this limit is reached. Default value is `50000`. | | `delta-padding-minutes` | integer | Insert the number of minutes to overlap when doing incremental load. When querying SAP for new or updated records, the extractor will read data starting from the last time the extractor ran minus this padding. This is to ensure some that there are some overlap in the queries and therefore no gaps. Default value is `1`. | | `chunk-size` | integer | Number of records to fetch in a single request to SAP. Default value is `1000`. | | [`state-store`](#extractor.state-store) | object | Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | ### `state-store` Part of [`extractor`](#extractor) configuration. Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time. | Parameter | Type | Description | | --------------------------------------- | ------ | ------------------------------------------------------------------------------------ | | [`raw`](#extractor.state-store.raw) | object | A RAW state store stores the extraction state in a table in CDF RAW. | | [`local`](#extractor.state-store.local) | object | A local state store stores the extraction state in a JSON file on the local machine. | #### `raw` Part of [`state-store`](#extractor.state-store) configuration. A RAW state store stores the extraction state in a table in CDF RAW. | Parameter | Type | Description | | ----------------- | ------- | ------------------------------------------------------------------------------------ | | `database` | string | **Required.** Enter the database name in CDF RAW. | | `table` | string | **Required.** Enter the table name in CDF RAW. | | `upload-interval` | integer | Enter the interval in seconds between each upload to CDF RAW. Default value is `30`. | #### `local` Part of [`state-store`](#extractor.state-store) configuration. A local state store stores the extraction state in a JSON file on the local machine. | Parameter | Type | Description | | --------------- | ------- | ----------------------------------------------------------------------- | | `path` | string | **Required.** Insert the file path to a JSON file. | | `save-interval` | integer | Enter the interval in seconds between each save. Default value is `30`. |