Edit

Share via

What is Apache Airflow Job?

  • Article

Note

Apache Airflow job is powered by Apache Airflow.

Apache Airflow job is the next generation of Azure Data Factory's Workflow Orchestration Manager. It's a simple and efficient way to create and manage Apache Airflow orchestration jobs, enabling you to run Directed Acyclic Graphs (DAGs) at scale with ease. It's the part of Fabric's Data Factory that empowers you with a modern data integration experience to ingest, prepare and transform data from a rich set of data sources for example, databases, data warehouse, Lakehouse, real-time data, and more.

When to use Apache Airflow job?

Apache Airflow jobs offers a managed service that enables users to create and manage Python-based DAGs (Directed Acyclic Graphs) for workflow orchestration, without worrying about the underlying infrastructure. If you have experience with Apache Airflow or you prefer code-centric approach, this option is ideal. In contrast, if you prefer a no-code solution for data orchestration, data pipelines offer a user-friendly alternative that doesn’t require managing or writing Python-based workflows.

Key Features

Microsoft Fabric hosted Apache Airflow job offer a range of powerful features, including:

Key Features Apache Airflow Job in Fabric Workflow Orchestration Manager in Azure Data Factory
Git sync Yes Yes
Enable AKV (Azure Key Vault) as backend Yes Yes
Install private package as requirement Yes Yes
Diagnostic logs and metrics No Yes
Blob Storage No Yes
Apache Airflow cluster IP address Yes Yes
Autoscale for managing production workload execution spikes Yes Partial
High Availability for mitigating outage/downtime Yes No
Deferrable Operators for suspending idle operators and free up workers Yes No
Pause and Resume TTL (Time to live) Yes No
SaaSified Experience - 10 secs to get started - Authoring DAGs - Fabric Free Trial Yes No

Region availability

  • Australia East
  • Australia Southeast
  • Brazil South
  • Canada Central
  • Canada East
  • Central India
  • Central US
  • East Asia
  • East US
  • East US 2
  • France Central
  • Germany West Central
  • Israel Central (coming soon)
  • Italy North (coming soon)
  • Japan East
  • Japan West (coming soon)
  • Mexico Central (coming soon)
  • North Europe
  • Norway East (coming soon)
  • Poland Central (coming soon)
  • South Africa North
  • South Central US
  • South India
  • Southeast Asia
  • Sweden Central
  • Switzerland North
  • Switzerland West
  • UAE North
  • UK South
  • UK West
  • West Europe
  • West US
  • West US 2
  • West US 3

Supported Apache Airflow versions

  • 2.10.5

Support Python version

  • 3.12

Note

Changing the Apache Airflow version within an existing Apache Airflow job is currently not supported. Instead, the recommended solution is to create a new Apache Airflow job with the desired version

Related Content