Apache Airflow: What It Is and How to Use It
Apache Airflow is an open-source platform for scheduling, running, and monitoring batch workflows — you write a workflow as Python code, give it a schedule, and Airflow runs each step in order, retries failures, and shows you what happened. It's the tool teams reach for when "a cron job and a shell script" stops being enough.
This is the practical version: what Airflow actually is, how the scheduler and DAGs work, the failure-handling that makes it worth the operational cost, when to use it instead of cron or a task queue, and a minimal walkthrough to define and run your first DAG. At the end, the concrete part — how we run dozens of DAGs in production behind paiddaily.io, and the two patterns those DAGs follow.
What Airflow is
Airflow is a scheduler that understands order, timing, and failure. You define a workflow as a Python DAG — a directed acyclic graph — where each node is a task and each edge is a dependency. Airflow reads your DAG, works out the order tasks have to run in, fires them on a schedule, and tracks the state of every run in a metadata database.
A few things it is not, because the confusion is common:
- It's not a streaming engine. Airflow runs batches on a schedule (every 5 minutes, hourly, daily). If you need sub-second event processing, you want Kafka or Flink, not Airflow.
- It's not a task queue. Celery and RQ run work as it arrives. Airflow runs work on a clock, with first-class concepts for "this run is for the 9am window" and "re-run last Tuesday."
- It's not a low-code tool. Workflows are Python files in a repo, version-controlled and code-reviewed like everything else. (If you want a visual canvas instead, that's a different trade-off — I wrote about when to reach for n8n vs writing the orchestrator yourself.)
Airflow is Apache 2.0 licensed and originated at Airbnb. You can self-host it or buy a managed version — more on that below.
What a DAG is
DAG stands for directed acyclic graph. In plain terms: a set of tasks with arrows showing which has to finish before the next can start, and no loops allowed (a task can't end up depending on itself).
In Airflow a DAG is just a Python file that declares:
- The tasks — the units of work (call an API, run a query, transform a file).
- The dependencies — which tasks run before which (
extract >> transform >> load). - The schedule — when the whole graph should run.
A classic example is an ETL job: extract from a source, transform the data, load it into a warehouse. Each step is a task; the arrows enforce the order; if extract fails, transform never runs. That ordering guarantee is the whole point of modeling it as a graph instead of a single script.
How the Airflow scheduler works
Airflow's scheduler is the long-running process that decides what runs and when — you start it as airflow scheduler, alongside the webserver, and it constantly scans your DAGs and creates a DAG run for each interval that's due.
This is what separates Airflow from a plain job scheduler like cron or a systemd timer: those fire one command at a time and forget it. Airflow's scheduler understands a whole graph of dependent jobs, tracks the state of every run, and knows the difference between "the 9am window" and "re-run last Tuesday." As a scheduling tool it sits closer to an orchestrator than to cron — which is why teams graduate to it when a folder of cron entries stops being enough.
Schedules are usually expressed as cron strings:
| Schedule | Cron | Runs |
|---|---|---|
| Every 5 minutes | */5 * * * * |
continuously, all day |
| Hourly | 0 * * * * |
top of every hour |
| Daily at 08:30 | 30 8 * * * |
once a day |
| Weekdays only | 0 9 * * 1-5 |
9am Mon–Fri |
Two ideas trip people up at first. First, Airflow schedules around data intervals — a daily DAG with start 2026-06-20 runs after the 20th completes, because it's processing that day's window. Second, catchup: if you deploy a DAG with a start date in the past, Airflow will by default try to run every interval it "missed." For most jobs you want catchup=False so it only runs from now forward.
Retries, dependencies, backfills, and failure handling
This is where Airflow earns its keep over a bare cron entry. Cron fires a command and forgets it. Airflow tracks the outcome and gives you the machinery around failure:
- Retries — set
retries=3andretry_delay, and a flaky API call gets re-attempted automatically before it's marked failed. - Dependencies — tasks only run when their upstream tasks succeed. A failed extract halts the load instead of loading garbage.
- Timeouts —
execution_timeoutkills a task that hangs, instead of letting it block the slot forever. - Failure callbacks —
on_failure_callbackfires a function when a task fails, so you can alert (Slack, Telegram, PagerDuty) instead of finding out from a stale dashboard. - Backfills — re-run a DAG over a historical date range with one command. If you fix a bug in a transform, you can replay last month's data through the corrected code:
airflow dags backfill -s 2026-05-01 -e 2026-05-31 my_dag.
The web UI ties it together: a grid of every run, green/red per task, logs one click away, and a button to clear-and-rerun a single failed task without touching the rest.
Airflow vs cron vs a task queue
The honest comparison, because reaching for Airflow too early is a real mistake:
| Cron | Task queue (Celery/RQ) | Airflow | |
|---|---|---|---|
| Triggered by | a clock | work arriving | a clock |
| Knows about order/dependencies | no | no (you wire it) | yes |
| Built-in retries/alerting | no | partial | yes |
| Backfill historical runs | no | no | yes |
| Monitoring UI | no | basic | yes |
| Operational weight | none | medium | heavy |
Use cron when you have a handful of independent jobs and a failure just means "it'll run again next time." A systemd timer or a crontab line is less to maintain than a scheduler, a webserver, and a metadata database.
Use a task queue when the work is event-driven — a user uploads a file, a webhook fires, a job needs to run now, not at the top of the hour.
Use Airflow when you have many scheduled jobs with dependencies between them, when per-job retry logic and failure alerting actually matter, and when you need to backfill or replay historical runs. The line, concretely: when the number of scheduled jobs exceeds what you can hold in your head. Below that, cron is fine. Above it, the UI, backfills, and per-task monitoring are worth the operational surface.
How to use it: a minimal DAG
Here's a complete, runnable DAG using the modern TaskFlow API. Drop this in your dags/ folder and Airflow picks it up.
from datetime import datetime, timedelta
from airflow.decorators import dag, task
@dag(
schedule="30 8 * * *", # daily at 08:30
start_date=datetime(2026, 1, 1),
catchup=False, # don't replay missed intervals
default_args={
"retries": 2,
"retry_delay": timedelta(minutes=5),
},
tags=["example"],
)
def daily_report():
@task
def extract() -> list[dict]:
# pull from an API or database
return [{"symbol": "ETH", "price": 3400}]
@task
def transform(rows: list[dict]) -> list[dict]:
for r in rows:
r["price_usd"] = round(r["price"], 2)
return rows
@task
def load(rows: list[dict]) -> None:
# write to your warehouse / Postgres
print(f"loaded {len(rows)} rows")
load(transform(extract())) # the arrows: extract >> transform >> load
daily_report()
The dependency graph is implied by how you pass the return values — extract() feeds transform() feeds load(). Airflow reads that and builds the DAG.
To run it locally without a full install:
# get a working Airflow in one process
pip install apache-airflow
export AIRFLOW_HOME=~/airflow
airflow standalone # starts scheduler + webserver, prints a login
Then open the UI at localhost:8080, find daily_report, unpause it with the toggle, and either wait for 08:30 or hit Trigger DAG to run it now. The grid view fills in green as tasks pass.
Self-hosting on Docker Compose vs managed
Once you're past a toy airflow standalone, you choose how to run it.
Self-hosted on Docker Compose is the common path for a small-to-mid stack. Apache ships an official docker-compose.yaml that brings up the scheduler, webserver, a Postgres metadata DB, and a worker. You mount your dags/ folder, set a few env vars, and docker compose up. You own the upgrades, the backups, and the box it runs on — but it sits right next to the rest of your services and costs nothing but the compute.
Managed Airflow — Astronomer, AWS MWAA, or GCP Cloud Composer — runs the control plane for you: autoscaling workers, upgrades, monitoring, the metadata DB. You write DAGs and push them; they keep the lights on. It's the right call when Airflow is mission-critical, the team is large, or you don't want to own scheduler ops. The trade-off is cost and a little less control over the runtime.
The rule of thumb: self-host while it's cheap to babysit; move to managed when the cost of it being down exceeds the cost of the platform.
How paiddaily.io uses it in production
That's the general tour. Here's the concrete part — Airflow running real workloads.
paiddaily.io is a consumer SaaS for full-time income traders. The dashboard shows DeFi positions, yield opportunities, market snapshots, and scoring analytics. All of that data has to stay fresh — some on 5-minute intervals, some daily — without making the API do the heavy lifting.
So Airflow runs dozens of DAGs covering price refreshes, pool syncs, market snapshots, ticker research, and scoring jobs. The split is clean: Airflow writes to Postgres, the API (FastAPI) reads from Postgres. Background work never touches the request path, so the API serves pure, fast reads. (The full architecture is its own post: the Airflow DAGs that run paiddaily.io.)
A sample of what's running:
- boros_snapshot_markets — every 5 minutes, fetches market snapshots from an external API, computes forecasts with catalyst awareness, archives raw data to a bronze layer, upserts the processed snapshot
- call_of_the_day — daily at 08:30 ET, gathers candidates, scores them with a multi-factor model, records the winner and runners-up
- pendle_market_research — runs a DSPy module against markets, persists the research output for the opportunity surface
- ticker_research_worker — enriches tickers with market cap, sector, and appearance data
- aero_pool_state_sync — syncs on-chain pool state from contracts on Base
- tastytrade_positions_sync — pulls positions from a brokerage API
Each DAG has retry logic, execution timeouts, and a failure callback so nothing fails silently.
Two patterns
Those DAGs fall into two patterns, and the split is worth naming because it comes up in any real Airflow codebase.
Direct domain calls
A large share of the DAGs call the same async domain functions the API uses, through a sync wrapper called run_domain:
from gl_paiddaily_common.domain_runner import run_domain
@task
def run_picker() -> dict:
return run_domain(
"features.common.jobs.call_of_the_day",
"run",
)
run_domain initializes an asyncpg connection pool, calls the async function, tears it down. Same business logic, separate process. This is the default when the domain code is already well-factored — the DAG is a thin scheduler over code that lives elsewhere.
Inline pipelines
The rest define the full pipeline in the DAG file itself — fetch from an external API, transform, write to Postgres. The Boros snapshot DAG is a good example: it reads active markets, fetches from the API, computes forecasts, writes raw data to a bronze archive, and upserts the snapshot. The logic is pipeline-shaped, not domain-shaped, so it lives in the DAG.
Both patterns end the same way: data lands in Postgres, the API reads it.
Adding a new job
The whole point of getting the structure right is that adding the next job is boring:
- Write the domain function. Pure async Python, returns a result.
- Create a DAG file. Use the
@dagand@taskdecorators. Callrun_domain()from the task (or write the pipeline inline). - Set the cron schedule.
- Wire failure alerting with
on_failure_callback.
Add a file, set a cron, done. That repeatability — not any single DAG — is the real reason Airflow earns its place in the stack once you're past a handful of jobs.
For the deeper architecture of how these pipelines fit together, see Apache Airflow in the resource notes and the full DAG breakdown.