# Cognite PostgreSQL Gateway

Use the Cognite PostgreSQL gateway together with your favored ETL tool or data warehouse solution to stream data into Cognite Data Fusion (CDF) via PostgreSQL. This service is useful for

  • data sources that can be accessed through existing ETL solutions that support writing to PostgreSQL, such as Microsoft Azure Data Factory, Talend and Pentaho.
  • extractors that can push data to PostgreSQL, but not CDF, such as ....

The Cognite PostgreSQL gateway can also be used for analyzing and visualizing data. This is useful for

  • applications with CDF data that support PostgreSQL as a source, but not CDF.
  • visualizing CDF data with a third-party dashboarding tool that does not support the OData service, such as Synapse.

Before you start configuring the Cognite PostgreSQL gateway, we recommend reading this article on planning your integration

In this article:

# Set up CDF

  1. Configure the Cognite Raw database for the ingestion of data with database name, table name and table columns (a good link?)

  2. Set up an service account for the integration and get the API key.

# Set up Cognite PostgreSQL Gateway

  1. Send a request to Cognite PostgreSQL gateway (https://skywarp.cognite.ai./create) with the CDF API key (see step 2 above), project name and the URL.
  2. The Cognite PostgreSQL gateway returns user name and password needed for configuring your ETL/data warehouse/connector. Note that the returned user has the same permissions to CDF as the provided API key. When the key is revoked, access is retracted.

No data is ever stored in the Cognite PostgreSQl gateway as the gateway sits in front of the CDF API and transpiles SQL to API calls towards CDF. Then the gateway uses the Foreign Data Wrapper functionality in PostgreSQL, where a foreign table in the database proxies the CDF data source.

(SHOW EXAMPLE?)

# Set up your PostgreSQL connector/ETL/data warehouse

  1. Configure your solution with user name, password, host name and database name (Does this apply to all solutions?)

  2. Set up to send integration information with integration external ID, integration name and any other relevant information

  3. Start the integration/Ddeploy to Cognite PostgreSQL gateway . See this example on how to set up Microsoft Azure Data Factory.

# Monitor the integrations

Use the Administer integration page to verify and monitor the integration. You can also edit the integration name, contact persons and description on this page.

Last Updated: 1/11/2021, 12:45:20 PM