Ga verder naar hoofdinhoud

Configuration settings

To configure the SAP extractor, you must create a configuration file. The file must be in YAML format.

You can use the sample minimal configuration file included with the extractor packages as a starting point for your configuration settings.

The configuration file contains the global parameter version, which holds the version of the configuration schema. This article describes version 1.

Tip

You can set up extraction pipelines to use versioned extractor configuration files stored in the cloud.

Using values from environment variables

The configuration file allows substitutions with environment variables. For example:

cognite:
secret: ${COGNITE_CLIENT_SECRET}

will load the value from the COGNITE_CLIENT_SECRET environment variable into the cognite/secret parameter. You can also do string interpolation with environment variables, for example:

url: http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}
Note

Implicit substitutions only work for unquoted value strings. For quoted strings, use the !env tag to activate environment substitution:

url: !env 'http://my-host.com/api/endpoint?secret=${MY_SECRET_TOKEN}'

Using values from Azure Key Vault

The DB extractor also supports loading values from Azure Key Vault. To load a configuration value from Azure Key Vault, use the !keyvault tag followed by the name of the secret you want to load. For example, to load the value of the my-secret-name secret in Key Vault into a password parameter, configure your extractor like this:

password: !keyvault my-secret-name

To use Key Vault, you also need to include the azure-keyvault section in your configuration, with the following parameters:

ParameterDescription
keyvault-nameName of Key Vault to load secrets from
authentication-methodHow to authenticate to Azure. Either default or client-secret. For default, the extractor will look at the user running the extractor, and look for pre-configured Azure logins from tools like the Azure CLI. For client-secret, the extractor will authenticate with a configured client ID/secret pair.
client-idRequired for using the client-secret authentication method. The client ID to use when authenticating to Azure.
secretRequired for using the client-secret authentication method. The client secret to use when authenticating to Azure.
tenant-idRequired for using the client-secret authentication method. The tenant ID of the Key Vault in Azure.

Example:

azure-keyvault:
keyvault-name: my-keyvault-name
authentication-method: client-secret
tenant-id: 6f3f324e-5bfc-4f12-9abe-22ac56e2e648
client-id: 6b4cc73e-ee58-4b61-ba43-83c4ba639be6
secret: 1234abcd

Base configuration object

ParameterTypeDescription
versioneither string or integerConfiguration file version
typeeither local or remoteConfiguration file type. Either local, meaning the full config is loaded from this file, or remote, which means that only the cognite section is loaded from this file, and the rest is loaded from extraction pipelines. Default value is local.
cogniteobjectThe cognite section describes which CDF project the extractor will load data into and how to connect to the project.
loggerobjectThe optional logger section sets up logging to a console and files.
metricsobjectThe metrics section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a Prometheus pushgateway, but you can also send metrics as time series in the CDF project.
saplistList of SAP instances to connect to
endpointslistList of endpoints to query
extractorobjectGeneral extractor configuration

cognite

Global parameter.

The cognite section describes which CDF project the extractor will load data into and how to connect to the project.

ParameterTypeDescription
projectstringInsert the CDF project name.
idp-authenticationobjectThe idp-authentication section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).
data-setobjectEnter a data set the extractor should write data into
extraction-pipelineobjectEnter the extraction pipeline used for remote config and reporting statuses
hoststringInsert the base URL of the CDF project. Default value is https://api.cognitedata.com.
timeoutintegerEnter the timeout on requests to CDF, in seconds. Default value is 30.
external-id-prefixstringPrefix on external ID used when creating CDF resources
connectionobjectConfigure network connection details

idp-authentication

Part of cognite configuration.

The idp-authentication section enables the extractor to authenticate to CDF using an external identity provider (IdP), such as Microsoft Entra ID (formerly Azure Active Directory).

ParameterTypeDescription
authoritystringInsert the authority together with tenant to authenticate against Azure tenants. Default value is https://login.microsoftonline.com/.
client-idstringRequired. Enter the service principal client id from the IdP.
tenantstringEnter the Azure tenant.
token-urlstringInsert the URL to fetch tokens from.
secretstringEnter the service principal client secret from the IdP.
resourcestringResource parameter passed along with token requests.
audiencestringAudience parameter passed along with token requests.
scopeslistEnter a list of scopes requested for the token
min-ttlintegerInsert the minimum time in seconds a token will be valid. If the cached token expires in less than min-ttl seconds, it will be refreshed even if it is still valid. Default value is 30.
certificateobjectAuthenticate with a client certificate

scopes

Part of idp-authentication configuration.

Enter a list of scopes requested for the token

Each element of this list should be a string.

certificate

Part of idp-authentication configuration.

Authenticate with a client certificate

ParameterTypeDescription
authority-urlstringAuthentication authority URL
pathstringRequired. Enter the path to the .pem or .pfx certificate to be used for authentication
passwordstringEnter the password for the key file, if it is encrypted.

data-set

Part of cognite configuration.

Enter a data set the extractor should write data into

ParameterTypeDescription
idintegerResource internal id
external-idstringResource external id

extraction-pipeline

Part of cognite configuration.

Enter the extraction pipeline used for remote config and reporting statuses

ParameterTypeDescription
idintegerResource internal id
external-idstringResource external id

connection

Part of cognite configuration.

Configure network connection details

ParameterTypeDescription
disable-gzipbooleanWhether or not to disable gzipping of json bodies.
status-forceliststringHTTP status codes to retry. Defaults to 429, 502, 503 and 504
max-retriesintegerMax number of retries on a given http request. Default value is 10.
max-retries-connectintegerMax number of retries on connection errors. Default value is 3.
max-retry-backoffintegerRetry strategy employs exponential backoff. This parameter sets a max on the amount of backoff after any request failure. Default value is 30.
max-connection-pool-sizeintegerThe maximum number of connections which will be kept in the SDKs connection pool. Default value is 50.
disable-sslbooleanWhether or not to disable SSL verification.
proxiesobjectDictionary mapping from protocol to url.

proxies

Part of connection configuration.

Dictionary mapping from protocol to url.

logger

Global parameter.

The optional logger section sets up logging to a console and files.

ParameterTypeDescription
consoleobjectInclude the console section to enable logging to a standard output, such as a terminal window.
fileobjectInclude the file section to enable logging to a file. The files are rotated daily.
metricsbooleanEnables metrics on the number of log messages recorded per logger and level. This requires metrics to be configured as well

console

Part of logger configuration.

Include the console section to enable logging to a standard output, such as a terminal window.

ParameterTypeDescription
leveleither DEBUG, INFO, WARNING, ERROR or CRITICALSelect the verbosity level for console logging. Valid options, in decreasing verbosity levels, are DEBUG, INFO, WARNING, ERROR, and CRITICAL. Default value is INFO.

file

Part of logger configuration.

Include the file section to enable logging to a file. The files are rotated daily.

ParameterTypeDescription
leveleither DEBUG, INFO, WARNING, ERROR or CRITICALSelect the verbosity level for file logging. Valid options, in decreasing verbosity levels, are DEBUG, INFO, WARNING, ERROR, and CRITICAL. Default value is INFO.
pathstringRequired. Insert the path to the log file.
retentionintegerSpecify the number of days to keep logs for. Default value is 7.

metrics

Global parameter.

The metrics section describes where to send metrics on extractor performance for remote monitoring of the extractor. We recommend sending metrics to a Prometheus pushgateway, but you can also send metrics as time series in the CDF project.

ParameterTypeDescription
push-gatewayslistList of prometheus pushgateway configurations
cogniteobjectPush metrics to CDF timeseries. Requires CDF credentials to be configured
serverobjectThe extractor can also be configured to expose a HTTP server with prometheus metrics for scraping

push-gateways

Part of metrics configuration.

List of prometheus pushgateway configurations

Each element of this list should be a the push-gateways sections contain a list of metric destinations.

ParameterTypeDescription
hoststringEnter the address of the host to push metrics to.
job-namestringEnter the value of the exported_job label to associate metrics with. This separates several deployments on a single pushgateway, and should be unique.
usernamestringEnter the credentials for the pushgateway.
passwordstringEnter the credentials for the pushgateway.
clear-aftereither null or integerEnter the number of seconds to wait before clearing the pushgateway. When this parameter is present, the extractor will stall after the run is complete before deleting all metrics from the pushgateway. The recommended value is at least twice that of the scrape interval on the pushgateway. This is to ensure that the last metrics are gathered before the deletion. Default is disabled.
push-intervalintegerEnter the interval in seconds between each push. Default value is 30.

cognite

Part of metrics configuration.

Push metrics to CDF timeseries. Requires CDF credentials to be configured

ParameterTypeDescription
external-id-prefixstringRequired. Prefix on external ID used when creating CDF time series to store metrics.
asset-namestringEnter the name for a CDF asset that will have all the metrics time series attached to it.
asset-external-idstringEnter the external ID for a CDF asset that will have all the metrics time series attached to it.
push-intervalintegerEnter the interval in seconds between each push to CDF. Default value is 30.
data-setobjectData set the metrics will be created under

data-set

Part of cognite configuration.

Data set the metrics will be created under

ParameterTypeDescription
idintegerResource internal id
external-idstringResource external id

server

Part of metrics configuration.

The extractor can also be configured to expose a HTTP server with prometheus metrics for scraping

ParameterTypeDescription
hoststringHost to run the prometheus server on. Default value is 0.0.0.0.
portintegerLocal port to expose the prometheus server on. Default value is 9000.

sap

Global parameter.

List of SAP instances to connect to

Each element of this list should be a configuration of an SAP source

Either one of the following options:

sap_netweaver_gateway

Part of sap configuration.

The SAP NetWeaver Gateway lets clients connect using the Open Data Protocol (OData).

Example:

type: odata
source-name: mys4hana
gateway-url: https://mys4hana.com/sap/opu/odata/sap/
client: '100'
username: ${SAP_USERNAME}
password: ${SAP_PASSWORD}
language: EN
ParameterTypeDescription
typealways odataRequired. Type of SAP source connection, set to odata for SAP OData sources.
source-namestringRequired. Enter a name for the source that will be used throughout the endpoints section and for logging. The name must be unique for each source in the configuration file.
gateway-urlstringRequired. Insert the SAP NetWeaver Gateway URL
clientstringRequired. Insert the SAP client number
usernamestringRequired. Enter the SAP username to connect to the SAP NetWeaver Gateway.
passwordstringRequired. Enter the SAP password to connect to the SAP NetWeaver Gateway
disable-sslbooleanDisable the SSL certificate verification towards the SAP destination. Default value is set to False.
connection_check_timeoutintegerEnter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is 60.
languagestringEnter the sap-language URL parameter. The default value is EN
certificatesobjectCertificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (ca-cert), public key (public-key), and private key (private-key).
proxyobjectHTTP and/or HTTPS proxies to be used when connecting to the SAP source system.
timezoneeither local or utcSpecify how the extractor should handle the source time zones. Default value is local.

certificates

Part of sap_netweaver_gateway configuration.

Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (ca-cert), public key (public-key), and private key (private-key).

ParameterTypeDescription
ca-certstringRequired. Enter the path to the CA certificate file.
public-keystringRequired. Enter the path to the public key file.
private-keystringRequired. Enter the path to the private key file.

proxy

Part of sap_netweaver_gateway configuration.

HTTP and/or HTTPS proxies to be used when connecting to the SAP source system.

ParameterTypeDescription
httpstringEnter the address of the HTTP proxy
httpsstringEnter the address of the HTTPS proxy.

sap_soap_source

Part of sap configuration.

The SAP extractor can connect to SAP instances using SOAP, such as for SAP ERP SOAMANAGER

Example:

type: soap
source-name: soap-funcloc
wsdl-url: https://myerp.com/sap/bc/srt/wsdl/sap/bc/srt/rfc/sap/test/100/test_funcloc/test_funcloc?sap-client=100
client: '100'
username: ${SAP_USERNAME}
password: ${SAP_PASSWORD}
language: EN
ParameterTypeDescription
typealways soapRequired. Type of SAP source connection, set to soap for SAP SOAP sources.
source-namestringRequired. Enter a name for the source that will be used throughout the endpoints section and for logging. The name must be unique for each source in the configuration file.
wsdl-urlstringRequired. Insert the SOAP WSDL URL related to the SAP ABAP webservice
strict-parserbooleanFlag to control how the extractor behave when parsing SOAP responses. Default value is set to True.
disable-sslbooleanDisable the SSL certificate verification towards the SAP destination. Default value is set to False.
clientstringRequired. Insert the SAP client number
usernamestringRequired. Enter the username to connect to the SAP Webservice.
passwordstringRequired. Enter the password to connect to the SAP Webservice
languagestringEnter the sap-language URL parameter. The default value is EN
certificatesobjectCertificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (ca-cert), public key (public-key), and private key (private-key).
connection_check_timeoutintegerEnter the timeout (in seconds) period the extractor should wait when doing connectivity tests towards the SAP source. Default value is 60.
session_timeoutintegerEnter the timeout (in seconds) period the extractor should use when fetching WSDL definitions from the SOAP source. Default value is 300.
timezoneeither local or utcSpecify how the extractor should handle the source time zones. Default value is local.

certificates

Part of sap_soap_source configuration.

Certificates needed for authentication towards SAP instance.

There are three certificates needed to perform the authentication: certificate authority (ca-cert), public key (public-key), and private key (private-key).

ParameterTypeDescription
ca-certstringRequired. Enter the path to the CA certificate file.
public-keystringRequired. Enter the path to the public key file.
private-keystringRequired. Enter the path to the private key file.

endpoints

Global parameter.

List of endpoints to query

Each element of this list should be a description of an endpoint to extract data from on one of the configured SAP sources.

ParameterTypeDescription
namestringRequired. Enter a name of this SAP endpoint that will be used to for logging. The name must be unique for each query in the configuration file.
source-namestringRequired. Enter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the sap section.
sap-servicestringRequired. Enter the name of the related SAP service. For odata endpoints, it's the SAP OData service. For soap endpoints, it's the service defined in the WSDL document.
sap-entitystringRequired. Enter the name of the related SAP entity. For odata endpoints, it's the name of the OData entity. For soap endpoints, it's the name of the SOAP operator as defined in the WSDL
destinationconfiguration for either RAW, Events, Assets, Time Series or FilesRequired. The destination of the data in CDF.
pagination-typeeither client, server or no-paginationOData pagination type when running full load (non incremental) from SAP. Default value is no-pagination.
sap-keylistEnter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using raw as the CDF destination.
selectlistEnter a list of fields to be selected from the SAP entity, using OData $select operation. This parameter is available only for SAP OData endpoints.
related-entitiesobjectSub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources.
attachmentsobjectExtraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations)
requeststringEnter the request to be sent to the SAP. This is a required parameter for soap endpoints. See the Requests section for more details.
incremental-fieldstringEnter the name of the field to be used as reference for the incremental runs. If this field is left out, the extractor will fetch full data loads every run.
incremental-field-formatstringEnter the date format of the incremental field so the extractor can convert it properly when running incremental queries. Example: %H:%M:%S is the date format for the date value 12:00:00. This is an optional parameter.
initial-startstringEnter the initial value to be used when running incremental requests. Mandatory parameter when running incremental loads in SOAP endpoints.
scheduleconfiguration for either Fixed interval or CRON expressionEnter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise.

Examples:
{'schedule': {'type': 'interval', 'expression': '1h'}}
{'schedule': {'type': 'cron', 'expression': '0 7-17 * * 1-5'}}
extract-schemaobjectIf included, the extractor will extract the SAP entity schema to CDF RAW
filterstringEnter the filter query string. The $filter system query option allows clients to filter a collection of resources from the target SAP OData endpoint. This is only relevant for odata sources.
soap-operationstringEnter the name of the SOAP operation defined in the WSDL configuration. For soap endpoints, this is a required parameter.
soap-portstringEnter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for soap endpoints.

Requests

The request parameter is part of the endpoints configuration, and is required soap endpoints.

SOAP requests

SAP ABAP Webservices are SOAP/based, meaning the requests to the SAP server must be in a valid XML format.

The SAP extractor expects this XML to be added as a string in the request parameter. This is an example of a valid XML request to a SAP ABAP Webservice generated from a SAP Function Module:

request: |
<FUNCLOC_LIST>
<item>
<FUNCTLOCATION>String 57</FUNCTLOCATION>
<FUNCLOC>String 58</FUNCLOC>
<LABEL_SYST>S</LABEL_SYST>
<DESCRIPT>String 60</DESCRIPT>
<STRIND>Strin</STRIND>
<CATEGORY>S</CATEGORY>
<SUPFLOC>String 63</SUPFLOC>
<PLANPLANT>Stri</PLANPLANT>
<MAINTPLANT>1010</MAINTPLANT>
<PLANGROUP>Str</PLANGROUP>
<SORTFIELD>String 67</SORTFIELD>
</item>
</FUNCLOC_LIST>
<MAINTPLANT_RA>
<item>
<SIGN>I</SIGN>
<OPTION>EQ</OPTION>
<LOW>1010</LOW>
<HIGH>1010</HIGH>
</item>
</MAINTPLANT_RA>

destination

Part of endpoints configuration.

The destination of the data in CDF.

Either one of the following options:

raw

Part of destination configuration.

The raw destination writes data to the CDF staging area (RAW). The raw destination requires the sap-key parameter in the endpoint configuration.

ParameterTypeDescription
typealways rawType of CDF destination, set to raw to write data to RAW.
databasestringEnter the CDF RAW database to upload data into. This will be created if it doesn't exist.
tablestringEnter the CDF RAW table to upload data into. This will be created if it doesn't exist.

events

Part of destination configuration.

The events destination inserts the resulting data as CDF events.

ParameterTypeDescription
typealways eventsType of CDF destination, set to events to write data to CDF Events.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Event
field-mapping

Part of events configuration.

Enter the fields of the SAP entity that corresponds to fields of an Event

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the event
startTimestringWhich field in the SAP entity containing the start time of the event
endTimestringWhich field in the SAP entity containing the end time of the event
descriptionstringWhich field in the SAP entity containing the description of the event
sourcestringWhich field in the SAP entity containing the source of the event
typestringWhich field in the SAP entity containing the type of the event
subTypestringWhich field in the SAP entity containing the subtype of the event

assets

Part of destination configuration.

The assets destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways assetsType of CDF destination, set to assets to write data to CDF Assets.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Asset
field-mapping

Part of assets configuration.

Enter the fields of the SAP entity that corresponds to fields of an Asset

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the asset
namestringWhich field in the SAP entity containing the name of the asset
parentExternalIdstringWhich field in the SAP entity containing the perent external ID of the asset
descriptionstringWhich field in the SAP entity containing the description of the asset
sourcestringWhich field in the SAP entity containing the source of the asset

time_series

Part of destination configuration.

The time_series destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways time_seriesType of CDF destination, set to time_series to write data to CDF Time Series.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of a time series data point
field-mapping

Part of time_series configuration.

Enter the fields of the SAP entity that corresponds to fields of a time series data point

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the time series to write to
timestampstringWhich field in the SAP entity containing the timestamp of the datapoint
valuestringWhich field in the SAP entity containing the value of the datapoint

files

Part of destination configuration.

The files destination writes data to the CDF staging area (Files)

ParameterTypeDescription
typealways filesType of CDF destination, set to files to write data to RAW.
sourcestringSets the 'Source' metadata field for the related files. This is an optional parameter.

sap-key

Part of endpoints configuration.

Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using raw as the CDF destination.

Each element of this list should be a string.

select

Part of endpoints configuration.

Enter a list of fields to be selected from the SAP entity, using OData $select operation. This parameter is available only for SAP OData endpoints.

Each element of this list should be a string.

Part of endpoints configuration.

Sub entities to be fetched from the main sap-entity. Feature is available for SOAP and OData sources.

ParameterTypeDescription
to_expandstringName of the value to be used in the OData $expand operation
sap-servicestringEnter the name of the related SAP service. For odata endpoints, it's the SAP OData service. For soap endpoints, it's the SOAP service defined in the WSDL document
sap-entitystringEnter the name of the related SAP entity. For odata endpoints, it's the name of the OData entity. For soap endpoints, it's the name of the SOAP object as defined in the WSDL.
source-namestringEnter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the sap section.
destinationconfiguration for either RAW, Events, Assets, Time Series or FilesThe destination of the data in CDF.
sap-keylistEnter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using raw as the CDF destination.
requeststringEnter the request to be sent to the SAP when using SOAP as a SAP source. This is a required parameter for soap endpoints. See the Requests section for more details.
parent-fieldstringEnter the name of the field from the parent entity to be used as input key when querying the related entity. This parameter is mandatory when using SOAP sources.
soap-operationstringEnter the name of the SOAP operation defined in the WSDL configuration. For soap endpoints, this is a required parameter.
soap-portstringEnter the name of the SOAP service port defined in the WSDL configuration. This parameter is only available for soap endpoints.

destination

Part of related-entities configuration.

The destination of the data in CDF.

Either one of the following options:

raw

Part of destination configuration.

The raw destination writes data to the CDF staging area (RAW). The raw destination requires the sap-key parameter in the endpoint configuration.

ParameterTypeDescription
typealways rawType of CDF destination, set to raw to write data to RAW.
databasestringEnter the CDF RAW database to upload data into. This will be created if it doesn't exist.
tablestringEnter the CDF RAW table to upload data into. This will be created if it doesn't exist.
events

Part of destination configuration.

The events destination inserts the resulting data as CDF events.

ParameterTypeDescription
typealways eventsType of CDF destination, set to events to write data to CDF Events.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Event
field-mapping

Part of events configuration.

Enter the fields of the SAP entity that corresponds to fields of an Event

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the event
startTimestringWhich field in the SAP entity containing the start time of the event
endTimestringWhich field in the SAP entity containing the end time of the event
descriptionstringWhich field in the SAP entity containing the description of the event
sourcestringWhich field in the SAP entity containing the source of the event
typestringWhich field in the SAP entity containing the type of the event
subTypestringWhich field in the SAP entity containing the subtype of the event
assets

Part of destination configuration.

The assets destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways assetsType of CDF destination, set to assets to write data to CDF Assets.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Asset
field-mapping

Part of assets configuration.

Enter the fields of the SAP entity that corresponds to fields of an Asset

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the asset
namestringWhich field in the SAP entity containing the name of the asset
parentExternalIdstringWhich field in the SAP entity containing the perent external ID of the asset
descriptionstringWhich field in the SAP entity containing the description of the asset
sourcestringWhich field in the SAP entity containing the source of the asset
time_series

Part of destination configuration.

The time_series destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways time_seriesType of CDF destination, set to time_series to write data to CDF Time Series.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of a time series data point
field-mapping

Part of time_series configuration.

Enter the fields of the SAP entity that corresponds to fields of a time series data point

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the time series to write to
timestampstringWhich field in the SAP entity containing the timestamp of the datapoint
valuestringWhich field in the SAP entity containing the value of the datapoint
files

Part of destination configuration.

The files destination writes data to the CDF staging area (Files)

ParameterTypeDescription
typealways filesType of CDF destination, set to files to write data to RAW.
sourcestringSets the 'Source' metadata field for the related files. This is an optional parameter.

sap-key

Part of related-entities configuration.

Enter a list of fields related to the SAP entity to be used as keys while ingesting data to CDF staging. This is a required parameter when using raw as the CDF destination.

Each element of this list should be a string.

attachments

Part of endpoints configuration.

Extraction of SAP attachments to CDF files. Feature is available only for OData endpoints (restricted to S/4HANA installations)

ParameterTypeDescription
destinationconfiguration for either RAW, Events, Assets, Time Series or FilesThe destination of the data in CDF.
object-keystringSAP primary key of the entity related to the attachment. This is a mandatory parameter.
business-object-typestringSAP Business Object Type related to the entity that contains the attachment. For example, BUS2038 is the standard business object type for SAP Maintenance Notifications. This is a mandatory parameter.
source-namestringEnter the name of the SAP source related to this endpoint. This must be one of the SAP sources configured in the sap section.
parallel-workersintegerNumber of parallel workers used to traverse and fetch download links from SAP attachments OData endpoint. Default value is 10.
session-timeoutintegerSet the timeout (in seconds) when listing files from SAP attachments OData endpoint. Default value is 10.

destination

Part of attachments configuration.

The destination of the data in CDF.

Either one of the following options:

raw

Part of destination configuration.

The raw destination writes data to the CDF staging area (RAW). The raw destination requires the sap-key parameter in the endpoint configuration.

ParameterTypeDescription
typealways rawType of CDF destination, set to raw to write data to RAW.
databasestringEnter the CDF RAW database to upload data into. This will be created if it doesn't exist.
tablestringEnter the CDF RAW table to upload data into. This will be created if it doesn't exist.
events

Part of destination configuration.

The events destination inserts the resulting data as CDF events.

ParameterTypeDescription
typealways eventsType of CDF destination, set to events to write data to CDF Events.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Event
field-mapping

Part of events configuration.

Enter the fields of the SAP entity that corresponds to fields of an Event

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the event
startTimestringWhich field in the SAP entity containing the start time of the event
endTimestringWhich field in the SAP entity containing the end time of the event
descriptionstringWhich field in the SAP entity containing the description of the event
sourcestringWhich field in the SAP entity containing the source of the event
typestringWhich field in the SAP entity containing the type of the event
subTypestringWhich field in the SAP entity containing the subtype of the event
assets

Part of destination configuration.

The assets destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways assetsType of CDF destination, set to assets to write data to CDF Assets.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of an Asset
field-mapping

Part of assets configuration.

Enter the fields of the SAP entity that corresponds to fields of an Asset

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the asset
namestringWhich field in the SAP entity containing the name of the asset
parentExternalIdstringWhich field in the SAP entity containing the perent external ID of the asset
descriptionstringWhich field in the SAP entity containing the description of the asset
sourcestringWhich field in the SAP entity containing the source of the asset
time_series

Part of destination configuration.

The time_series destination inserts the resulting data as CDF assets.

ParameterTypeDescription
typealways time_seriesType of CDF destination, set to time_series to write data to CDF Time Series.
field-mappingobjectEnter the fields of the SAP entity that corresponds to fields of a time series data point
field-mapping

Part of time_series configuration.

Enter the fields of the SAP entity that corresponds to fields of a time series data point

ParameterTypeDescription
externalIdstringRequired. Which field in the SAP entity containing the external ID of the time series to write to
timestampstringWhich field in the SAP entity containing the timestamp of the datapoint
valuestringWhich field in the SAP entity containing the value of the datapoint
files

Part of destination configuration.

The files destination writes data to the CDF staging area (Files)

ParameterTypeDescription
typealways filesType of CDF destination, set to files to write data to RAW.
sourcestringSets the 'Source' metadata field for the related files. This is an optional parameter.

schedule

Part of endpoints configuration.

Enter the schedule for when this query should run. Make sure not to schedule runs too often, but leave some room for the previous execution to be done. Required when running in continuous mode, ignored otherwise.

Either one of the following options:

Examples:

schedule:
type: interval
expression: 1h
schedule:
type: cron
expression: 0 7-17 * * 1-5

fixed_interval

Part of schedule configuration.

ParameterTypeDescription
typealways intervalRequired. Type of time interval configuration. Use interval to configure a fixed interval.
expressionstringRequired. Enter a time interval, with a unit. Avaiable units are s (seconds), m (minutes), h (hours) and d (days).

Examples:
45s
15m
2h

cron_expression

Part of schedule configuration.

ParameterTypeDescription
typealways cronRequired. Type of time interval configuration. Use cron to configure CRON schedule.
expressionstringRequired. Enter a CRON expression. See crontab.guru for a guide on writing CRON expressions.

Example:
*/15 8-16 * * *

extract-schema

Part of endpoints configuration.

If included, the extractor will extract the SAP entity schema to CDF RAW

ParameterTypeDescription
typealways rawType of CDF destination, always set to raw
databasestringEnter the CDF RAW database to upload data into. This will be created if it doesn't exist.
tablestringEnter the CDF RAW table to upload data into. This will be created if it doesn't exist.

extractor

Global parameter.

General extractor configuration

ParameterTypeDescription
modeeither continuous or singleExtractor mode. Continuous runs the configured queries using the schedules configured per query. Single runs the queries once each. Default value is single.
parallelismintegerMaximum number of parallel queries. Default value is 10.
upload-queue-sizeintegerMaximum size of upload queue. Upload to CDF will be triggered once this limit is reached. Default value is 50000.
delta-padding-minutesintegerInsert the number of minutes to overlap when doing incremental load. When querying SAP for new or updated records, the extractor will read data starting from the last time the extractor ran minus this padding. This is to ensure some that there are some overlap in the queries and therefore no gaps. Default value is 1.
chunk-sizeintegerNumber of records to fetch in a single request to SAP. Default value is 1000.
state-storeobjectInclude the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time.

state-store

Part of extractor configuration.

Include the state store section to save extraction states between runs. Use this if data is loaded incrementally. We support multiple state stores, but you can only configure one at a time.

ParameterTypeDescription
rawobjectA RAW state store stores the extraction state in a table in CDF RAW.
localobjectA local state store stores the extraction state in a JSON file on the local machine.

raw

Part of state-store configuration.

A RAW state store stores the extraction state in a table in CDF RAW.

ParameterTypeDescription
databasestringRequired. Enter the database name in CDF RAW.
tablestringRequired. Enter the table name in CDF RAW.
upload-intervalintegerEnter the interval in seconds between each upload to CDF RAW. Default value is 30.

local

Part of state-store configuration.

A local state store stores the extraction state in a JSON file on the local machine.

ParameterTypeDescription
pathstringRequired. Insert the file path to a JSON file.
save-intervalintegerEnter the interval in seconds between each save. Default value is 30.