Before you start
- Check the server requirements for the extractor.
- Assign access capabilities in a CDF project.
- Set up a Windows Update schedule. Note that the update may reboot the machine, causing extractor downtime.
- To create and monitor the extractor as an integration (public preview), see Set up an on-premises integration. You can also navigate to Data fusion > Integrate > Extractors to download the Windows executable file.
- Create a configuration file according to the configuration settings. The file must be in YAML format.
Trusting internal and self-signed certificates
The extractor uses HTTPS to connect to file sources, such as SharePoint Online, Azure Blob Storage, or Documentum, and to the Cognite Data Fusion (CDF) API. If your file source uses an internal or self-signed certificate, or if a corporate proxy inspects outbound traffic with certificates signed by an internal certificate authority (CA), add the certificate to the extractor’s certificate bundle so connections between the source system, the extractor host, and CDF remain trusted. IfREQUESTS_CA_BUNDLE points only at the corporate CA, endpoints that still present a public certificate chain are not trusted. SharePoint Online and Microsoft Graph requests do not log a certificate error. The extractor keeps retrying those requests.
Create a combined certificate bundle
Locate the public CA bundle
certifi library:Copy the bundle to a persistent directory
cacert.pem:- Windows:
C:\Cognite\certificates\cacert.pem - Linux:
/opt/cognite/certificates/cacert.pem
Add your certificate
- Obtain the certificate of the file source or proxy in PEM format.
- Open the certificate in a text editor.
- Open the
cacert.pemfile you saved. - Prepend the certificate contents to the top of the file.
- Save the updated file.
cacert.pem file contains the public CAs and your internal or self-signed certificate.Configure the extractor
Set REQUESTS_CA_BUNDLE
REQUESTS_CA_BUNDLE to the path of the updated cacert.pem file. Set it at the machine level or in the service unit so the extractor service picks it up.- Windows
- Linux
- macOS
.pem file.$env:REQUESTS_CA_BUNDLE="C:\Cognite\certificates\cacert.pem" in PowerShell or export REQUESTS_CA_BUNDLE=/opt/cognite/certificates/cacert.pem in Bash, and run the extractor from that same shell.Restart the extractor
REQUESTS_CA_BUNDLE for the executable or service even when the same script works without it.Sources that use other settings
REQUESTS_CA_BUNDLE covers the CDF connection and most file sources, including SharePoint Online, Azure Blob Storage, Azure Data Lake Storage, Google Cloud Storage, and AWS S3. Two sources read their certificate settings elsewhere:
Run as a Windows standalone executable file
Download the executable
file_extractor-{VERSION}-win32.exe file via the download links available from the Cognite File extractor section on the Extract data page in the CDF user interface.Save the file
Run the file
config.yml and saved in the same folder as the executable file:Install and run the extractor
- Windows
- Linux
Install using Windows installer
Download the installer
file-extractor-{VERSION}-win32.msi file via the download links available from the Cognite File extractor section on the Extract data page in the CDF user interface.Save the file
Navigate to the folder
Install the extractor
Run as standalone executable
Download the executable
file_extractor-{VERSION}-win32.exe file via the download links available from the Cognite File extractor section on the Extract data page in the CDF user interface.Save the file
Run the file
config.yml and saved in the same folder as the executable file:Run as Windows service
Download the service executable
file-extractor-service-{VERSION}-win32.exe file via the download links available from the Cognite File extractor section on the Extract data page in the CDF user interface.Save the file
config.yaml. This must be the exact name of the file.Install the service
Open the Services app
Configure the service
Adjust settings