> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Use data workflows

> Create, run, and monitor data workflows using Cognite API or Cognite Python SDK. Use Data workflows in the CDF user interface to automate and manage workflow tasks and process runs.

## Before you start

To add the necessary capabilities for data workflows, see [assign capabilities](/cdf/access/guides/capabilities#data-workflows).

To set up email notifications for workflows, you also need the `subscribesignals:read` and `subscribesignals:write` [capabilities](/cdf/access/guides/capabilities#data-workflows).

<Tip>
  You can assign a workflow to a [data set](/cdf/data_governance/concepts/datasets) when you create a workflow. All `read` or `write` operations on the workflow or related resources, such as versions, runs, tasks, and triggers, need access to this data set.
</Tip>

## Create data workflows

Use <span class="ui-element">Data workflows</span> in the CDF user interface to automate and manage tasks and processes runs.

<Steps>
  <Step title="Navigate to Data workflows">
    Navigate to <span class="ui-element">Data fusion</span> > <span class="ui-element">Data workflows</span>.
  </Step>

  <Step title="Create and configure workflow">
    Select <span class="ui-element">+ Create workflow</span> and follow the wizard to start building your workflow:

    * Add [workflow tasks](/cdf/data_workflows/task_types#task-types).

    * Use the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor) when you configure **Transformation**, **Function**, or **Agent** tasks that accept [dynamic references](/cdf/data_workflows/task_types#dynamic-references).

    * Create [workflow triggers](/cdf/data_workflows/triggers).

    * Switch between editing the workflow and viewing the workflow run history.

    * View or modify workflow versions.

    * Run the workflow, with or without input data.
  </Step>
</Steps>

<Tip>
  **Control concurrent runs for predictable processing**

  You can set the maximum number of concurrent runs for each workflow to ensure predictable and consistent data processing. Open **Workflow details** to enter the maximum number of concurrent runs. When the limit is reached, new workflow runs will queue until existing runs complete.
</Tip>

## Run data workflows

To run a workflow, call the API endpoint [`/run`](/api-reference/concepts/20230101/workflow-executions), and provide `workflowExternalId` and `version`.

### Authenticate a workflow

To start a workflow run, the workflow requires a [nonce](/cdf/access/concepts/authentication_flows_oidc#on-behalf-of-flow). This is a temporary token used for authentication when the workflow triggers the processes defined by its tasks. A nonce can be retrieved from the [Sessions API](/api-reference/concepts/20230101/function-calls) when creating a session. The session will be created behind the scenes when triggering a workflow using the Python SDK.

All tasks in the workflow execution use the same nonce for authentication, except for transformation tasks. These tasks are configured with `useTransformationCredentials` set to `true`. The transformation job uses the credentials specified in the transformation. When a trigger creates a workflow run, the nonce configured for the trigger will be passed to the workflow run.

<Info>
  Make sure that the nonce has the necessary capabilities to perform the tasks in the workflow.
</Info>

### Define workflow input data

You can define custom `input` data as part of the workflow run. This means workflows can behave differently based on the input ([parametric workflows](/cdf/data_workflows/task_types#make-workflows-parametric)). The input data is accessible throughout the workflow using [dynamic references](/cdf/data_workflows/task_types#dynamic-references).

<Info>
  Don't use input data to pass large amounts of data to the workflow. For large data sets, we recommend using CDF's data stores and setting the tasks to directly read from and write to these. See [Limits and restrictions](/cdf/data_workflows/limits_and_restrictions_workflows) for input size limits.
</Info>

The workflow input can come from multiple sources:

* **Manual execution input**: Data you provide when manually triggering a workflow.
* **Trigger input**: Static data defined when you create triggers.
* **Trigger-generated input**: Dynamic data collected by triggers, such as data modeling triggers, add `items` to arrays.
* **System input**: Metadata automatically added by the system, such as `version`.

### Add  metadata

A workflow run can also have custom, application-specific metadata, which is separate from the input data. Use metadata to track execution context, add operational information, and facilitate debugging and monitoring.

## Monitor workflows

Track workflow health and receive alerts about execution issues to ensure your workflows run smoothly.

### Workflow execution statuses

During execution, each workflow progresses through different statuses that indicate its current state:

| Status       | Description                                                                    |
| ------------ | ------------------------------------------------------------------------------ |
| `RUNNING`    | The workflow is currently running.                                             |
| `COMPLETED`  | All tasks have completed successfully.                                         |
| `FAILED`     | The workflow failed to complete due to an error and could not continue to run. |
| `TIMED_OUT`  | The workflow exceeded its maximum execution time.                              |
| `TERMINATED` | The workflow was manually canceled or stopped by another workflow or process.  |

### Email notifications

Receive email alerts when workflows fail, time out, or when triggers fail to execute. Users with the appropriate [access capabilities](/cdf/access/guides/capabilities#data-workflows) can configure notifications for any user, while other users can only configure their own settings.

To configure notifications, select the workflow external ID in the top bar and choose <span class="ui-element">Notification settings</span>. If you're an admin, select which users should receive notifications. If you're a non-admin user, you can configure your own notification settings.

You can also add custom email addresses to receive notifications, for example shared inboxes, or channels in other systems like Slack or Microsoft Teams.

Select which events should trigger notifications:

* **Workflow execution failed**: Alerts when a workflow run fails
* **Workflow execution timed out**: Alerts when a workflow exceeds its user-defined maximum execution time
* **Trigger attempt failed**: Alerts when a trigger fails to execute

<Info>
  Each user can have different notification preferences. Return to notification settings to update or remove users.
</Info>

## Task input and output

Each task receives a task definition as input during a workflow run. Different outputs are returned depending on the task type:

| Task type          | Output                                                                                                                                        |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Transformation** | `jobId` of the transformation job in CDF.                                                                                                     |
| **Function**       | `callId`, `functionId`, and `response` of the [Cognite Functions](/cdf/functions).                                                            |
| **Simulation**     | `runId`, `logId`, and `statusMessage` of the [simulation run in CDF](/cdf/integration/guides/simulators/simulator_trigger).                   |
| **CDF**            | `statusCode` and `response` (body) of the API call.                                                                                           |
| **Dynamic**        | `dynamicTasks`, an array of dynamically generated tasks based on the input data. Each task in this array has its own `externalId` and `type`. |

These outputs can be referenced by subsequent tasks using [dynamic references](/cdf/data_workflows/task_types#dynamic-references), enabling data flow between tasks. For supported tasks in the user interface, use the [reference picker](/cdf/data_workflows/task_types#reference-picker-in-the-editor). For dynamic tasks, refer to the `externalId` of the dynamically generated tasks to access their outputs.

<Warning>
  **Using client secrets as input to workflows and tasks**

  Some tasks in a workflow run may need access to client secrets, client credentials, or other confidential information. <span class="ui-element">Data workflows</span> currently don't support secure storage of client secrets as input to workflows or tasks.

  We recommend leveraging the capabilities for secure storage of secrets in [Cognite Functions](/cdf/functions), combined with a Function task in the workflow.
</Warning>

## Automate workflows with triggers

Use triggers to **automate your data workflow runs**. Run workflows at regular intervals for scheduled automation, for instance for daily reports or maintenance tasks or to  react to data changes in real-time. Monitor specific conditions and automatically start workflow executions when the conditions are met.

Triggers also automatically handle groups of related data changes (batch processing). For details about trigger types, configuration, and how they pass data to workflows, see [Triggers for data workflows](/cdf/data_workflows/triggers) and [API workflow triggers](/api-reference/concepts/20230101/workflow-triggers).

## Error handling and retries

Workflows provide built-in error-handling mechanisms:

* **Task-level retries**: Each task can be configured with retry attempts.

* **Failure policies**: Tasks can be marked as optional (`skipTask`) or required (`abortWorkflow`).

* **Compensation flows**: Use subsequent tasks to handle failures or cleanup

* **Manual intervention**: Use Function tasks with `isAsyncComplete=true` to create approval points where the workflow waits for human review and manual task completion.

For details about task configuration and error handling, see [Tasks in data workflows](/cdf/data_workflows/task_types).