ext_onelake. Transformation results are still written to CDF destinations such as data models.
Fabric remains the system of record for the source tables. Each run reads the current Delta table state over ABFSS (Azure Blob File System Secure).
What this path does
How it works
- Register a Fabric OneLake integration as an external data source (credentials plus workspace and lakehouse IDs).
- Reference a Delta table in transformation SQL with
ext_onelake(...). - Shape the data with Spark SQL.
- Write the result to a CDF destination such as a data model.
Requirements
- Fabric Lakehouse Delta tables under
Tables/(optional single schema folder). - Microsoft Entra ID service principal with at least Contributor on the Fabric workspace or lakehouse (Viewer is not enough for OneLake API access).
- CDF capabilities for external data sources and Transformations.
Incremental loading with is_new
Useis_new() so Transformations process only new or changed rows. Your Fabric Delta table must include a TIMESTAMP or epoch-millisecond LONG column that your pipeline updates when a row changes. Fabric does not maintain this column. Without it, each run reads the full table.
For syntax and examples, see Read Fabric OneLake data.
Known limitations
Deleted or updated rows can reappear
If the Fabric table uses deletion vectors (the Fabric Spark default),ext_onelake can return rows that Fabric Spark SQL and the SQL analytics endpoint already hide. MERGE, UPDATE, and DELETE mark rows as removed. ext_onelake does not apply those marks.
For purge, copy, and de-duplicate options, see Extra, deleted, or updated rows appear in results.
Next steps
Set up an external data source
Register Fabric OneLake credentials and location with the API.
Read Fabric OneLake data
Query with
ext_onelake and write to a CDF destination.