> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Cognite Data Fusion (CDF) is an industrial DataOps platform. Prefer English docs unless the user asks for another locale.
> Follow /_llms/ index links recursively until you reach page URLs ending in .md. Fetch those Markdown twins instead of HTML.
> For REST APIs, default to calendar version 20230101 (stable). Use 20230101-beta or 20230101-alpha only if the user is on a preview API. Cite paths under /api-reference/concepts/<version>/ and the Cdf-Version header. Prefer the Python SDK (cognite-sdk) unless the user is in JavaScript/TypeScript.
> Start here by task: platform overview /cdf/index.md; data modeling /cdf/dm/index.md; data integration /cdf/integration/index.md; access /cdf/access/index.md; CDF Toolkit /cdf/deploy/cdf_toolkit/index.md; REST quickstart /dev/quickstart.md; Python SDK https://cognite-sdk-python.readthedocs-hosted.com/en/stable/; Atlas AI /cdf/atlas_ai/concepts/index.md; Flows apps /cdf/flows/index.md; MCP and IDE setup /dev/guides/ide_ai_integration.md.
> For implementation work, also load /skill.md.

# About transformations

> Learn about data transformation in Cognite Data Fusion (CDF) and how to use built-in transformation tools to reshape, enrich, contextualize, and analyze data quality in your data integration workflow.

## Transformation architecture and purpose

A transformation specifies how source data is mapped and written to a target structure in CDF.

Transformations can enrich data with other sources or calculations, contextualize it by matching related objects, and check data quality before it reaches downstream users. A common pattern is to read staged data from CDF RAW and write it to a structured target such as a data model. You can also read registered **external data sources**—for example Microsoft Fabric OneLake Delta tables with `ext_onelake`—without staging a full copy in CDF RAW first. In both cases, transformation results still land in CDF destinations. This helps you deliver consistent, queryable data to apps and workflows, for example when you standardize equipment metadata before it is used in dashboards.

```mermaid theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
flowchart LR
  subgraph Sources
    A[Source systems]
    A2[External data sources]
  end

  subgraph Staging
    B[RAW staging]
  end

  subgraph Processing
    C[[Transformations]]
    C1[(Spark SQL)]
    C --> C1
  end

  subgraph Targets
    D[Data models]
    E[Sequences]
    F[Files]
  end

  A --> B --> C
  A2 --> C
  C --> D
  C --> E
  C --> F

  classDef stage fill:#EEF6FF,stroke:#3B82F6,stroke-width:1px;
  classDef process fill:#F0FDF4,stroke:#16A34A,stroke-width:1px;
  classDef target fill:#FFF7ED,stroke:#EA580C,stroke-width:1px;
  class B stage;
  class C,C1 process;
  class D,E,F target;
```

Transformations are developer-centric tools for data engineers and developers who define schemas, write SQL, and manage pipelines. **CDF Transformations run on a managed Spark SQL engine.** You express logic in SQL (or map fields in the UI), and CDF handles scheduling, scaling, and access to CDF data sources and registered external data sources.

* Start with SQL and use transformations when your logic is declarative and best expressed as set-based operations across tables.

* Use [CDF Functions](/cdf/functions) when you need Python logic, external libraries, or custom API calls.

* Use the [Cognite Toolkit](/cdf/deploy/cdf_toolkit) when you want to manage transformations as code. The Toolkit lets you define transformations, schedules, and notifications in YAML (with optional SQL files) and deploy them through CI/CD.

* You can also run transformations using the [Cognite API](/api-reference/concepts/20230101/transformations), and the [Cognite Python SDK](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/transformations.html).

<Warning>
  Avoid transformations for high-volume or low-latency writes such as high-frequency datapoints. For these scenarios, prioritize direct ingestion pipelines or [Files](/api-reference/concepts/20230101/files) to keep latency and throughput within system limits.
</Warning>

## Target selection and operational constraints

Decide where data should land early, and avoid defaulting to the CDF staging area ([RAW](/dev/concepts/resource_types/raw)) as the final destination. Use RAW for staging, then write to a target that matches how the data will be used.

<Tip>
  Treat RAW as a staging layer and align the target with how the data will be used.
</Tip>

<CardGroup cols={2}>
  <Card title="Create a transformation" icon="gear" href="/cdf/integration/guides/transformation/transformations">
    Define and run your first transformation in CDF.
  </Card>

  <Card title="SQL syntax and functions" icon="code" href="/cdf/integration/guides/transformation/write_sql_queries">
    Reference for SQL syntax and custom functions.
  </Card>

  <Card title="Fabric OneLake in Transformations" icon="database" href="/cdf/integration/concepts/transformation/about_fabric_onelake_transformations">
    Read Fabric OneLake Delta tables without a full RAW staging copy.
  </Card>

  <Card title="Set up a Fabric OneLake source" icon="plug" href="/cdf/integration/guides/transformation/set_up_fabric_onelake_connection">
    Register an external data source for OneLake (API).
  </Card>
</CardGroup>