Data Engineering moderneNovember 2, 2026

Airflow: orchestrating pipelines without freezing your data architecture

Behind a stack of modern data tools, the orchestrator often becomes the point of tension. A look at Airflow's architecture and what it means for teams industrializing their pipelines.

When a modern data stack comes together, attention often focuses on the compute engines — Snowflake, BigQuery, Databricks — or on the dbt model. The orchestrator stays in the background… until the day a silently failing pipeline, a cross-dependency, or a security question forces you to understand what is actually happening under the hood. For a CTO or data team lead, the initial choice of Airflow is rarely the problem; it is how its architecture is deployed and governed.

The official Apache Airflow documentation provides a useful framework for this reflection. It describes a platform that lets you build and run workflows as DAGs (Directed Acyclic Graphs), where each task represents a unit of work: fetching data, running analysis, triggering an external system. Airflow remains "agnostic to what you're running," the architecture documentation notes. It happily orchestrates a Python operator or a shell command, which explains its central place in heterogeneous environments.

But this flexibility has an architectural cost. A minimal Airflow installation relies on several required components: a scheduler that triggers scheduled workflows and submits tasks to the executor, a Dag processor that parses Dag files and serializes them into the metadata database, a Dag bundle that holds the DAG versions, an API Server that exposes the UI and REST API, and a metadata database (usually PostgreSQL or MySQL). In a simple deployment, everything runs on a single machine. In a distributed environment, these components can be spread across multiple machines with distinct security perimeters.

What this separation changes for security

The most structuring point for a technical decision maker is not the number of components, but the isolation logic they enable. In a distributed deployment, the API Server does not have direct access to the Dag bundles: the code shown in the Code tab of the UI is read from the metadata database. The API Server cannot execute any code submitted by the Dag author. It can only execute code installed as a package or plugin by the Deployment Manager.

This distinction between roles — Deployment Manager, Dag author, Operations User — has concrete implications. The person writing DAGs is not necessarily the person managing the infrastructure, and the person monitoring execution should not be able to modify the code. The official documentation insists: in a distributed deployment, it is important to consider the security aspects of the components. The Dag processor and workers need access to the Dag bundles; the scheduler and triggerer read from the metadata database and do not require direct access to the bundles.

For a team industrializing pipelines around dbt or data contracts, this separation is not theoretical. It shapes how you answer practical questions: who can modify a critical pipeline in production? How do you guarantee that the code executed on a worker matches the version reviewed upstream? How do you prevent a compromised dependency in a DAG from gaining access to the entire metadata database?

Implications for task execution

Another architectural decision, often underestimated, concerns execution itself. A worker never runs task code in its own process. For every task instance, it starts a new subprocess, supervises it, and tears it down once the task finishes. Whether the task is written in Python, Java, or Go, each concurrent instance adds a cost in subprocesses and memory. Sizing a worker therefore means budgeting for the peak number of tasks running at once, not just the load of the worker process itself.

This behavior has consequences for operational choices. If a team runs dbt commands on Airflow workers with the CeleryExecutor or KubernetesExecutor, the concurrency question arises differently than on a simple local server. The documentation notes that nothing is pooled or reused between task instances: N concurrent tasks cost N subprocesses, plus anything the task code itself spawns.

This is also an argument for thinking about DAG granularity. Multiplying small tasks is not neutral in resources. Conversely, grouping too much logic into a single operator reduces graph readability and complicates recovery in case of failure. The right balance depends on the data monitoring policy, execution frequency, and traceability requirements.

What to take away for a modern data stack

Airflow does not impose a single model. The documentation describes a continuum, from a basic single-machine deployment to a distributed architecture with separate roles and isolated security perimeters. What the tool does not solve is the governance around this continuum.

For a company assembling Snowflake or BigQuery, dbt, and Airflow, the structuring choices sit at three levels: the distribution of roles between those who deploy, those who write, and those who operate; the DAG versioning policy, notably via bundle backends like Git to avoid version inconsistencies between the Dag processor and workers; and worker sizing based on real peak concurrency.

These topics are not implementation details. They determine whether the orchestrator remains a trusted tool or becomes a blind spot in the data chain. To explore how these choices fit into a broader data platform, you can review our services or contact us.

Sources

Share this article

Read also