ext_snowflake. Transformation results are still written to CDF destinations such as data models.
Snowflake remains the system of record for the source tables. Each run reads the current table state on your Snowflake warehouse. CDF generates an RSA key pair, stores the private key encrypted, and returns only the public key.
What this path does
How it works
- Register a Snowflake integration as an external data source. CDF returns a public key.
- Send that public key to your Snowflake admin. The admin registers it on the Snowflake service user.
- Reference a table in transformation SQL with
ext_snowflake(...). - Shape the data with Spark SQL.
- Write the result to a CDF destination such as a data model.
Requirements
- A Snowflake service user and service role with
USAGEon the database, schema, and warehouse, andSELECTon the tables. Standard tables, dynamic tables, and Snowflake-managed Iceberg tables are supported. - A warehouse that runs the reads.
- Network access from CDF to the Snowflake account. If the account uses a network policy, contact Cognite Support for the CDF egress IP addresses. Your Snowflake admin allowlists those addresses so CDF can reach the account.
- CDF capabilities for external data sources and Transformations.
Incremental loading with is_new
Useis_new() so Transformations process only new or changed rows. Your Snowflake table must include a TIMESTAMP or epoch-millisecond LONG column that your pipeline updates when a row changes. Snowflake does not maintain this column. Without it, each run reads the full table.
An incremental read does not pick up deleted rows. See Known limitations.
For syntax and examples, see Read Snowflake data.
Known limitations
Deleted rows are not removed
Rows deleted in Snowflake do not appear in an incremental read, and CDF does not remove the matching instances. For a way to reflect deletes, see Deleted rows stay in CDF.Write-back is not supported
ext_snowflake is read-only. Write-back is not supported.
Iceberg files are read in Snowflake
This connection reads Snowflake tables, including Snowflake-managed Iceberg tables. CDF does not read the Iceberg files directly. Snowflake reads the files, and CDF receives the query results.A RAW destination adds a staging copy
Prefer a data model destination. Writing the result to RAW adds the staging copy this path is meant to skip.Next steps
Set up an external data source
Register the Snowflake connection in Integrations or with the API. Your Snowflake admin registers the public key.
Read Snowflake data
Query with
ext_snowflake and write to a CDF destination.