Project Overview
Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows. You write pipelines as Python code — directed acyclic graphs (DAGs) of tasks — and Airflow's scheduler executes them on a schedule, retries what fails, and gives you a web UI that shows exactly what ran, when it ran, and why it broke.
Airflow is a top-level Apache Software Foundation project under the Apache-2.0 license, so you get permissive commercial use with no relicensing surprises. It is also one of the most mature options in workflow orchestration, with extensive official documentation and an active developer community.
Be clear about what Airflow is not. It orchestrates work; it does not process data. If a task pulls a terabyte through the worker's own memory, that's your code making a mistake Airflow won't stop. Use it to tell Spark, dbt, or your warehouse what to run and when — and keep heavy data movement inside the systems built for it.
It is also batch-first. The scheduler thinks in runs and intervals, plus data-aware triggers in recent versions. If you need sub-second, event-driven execution, pick a different tool rather than fighting the scheduler.
Key Challenges Addressed
You usually reach for Airflow after cron stops scaling. The concrete failures it targets:
- Dependency-blind scheduling. Cron fires jobs at fixed times and hopes upstream data arrived. Airflow models dependencies explicitly — task B runs only after task A succeeds — so a late upstream can't silently feed stale data into a downstream report.
- No retries, no alerting. A cron job that dies at 3 a.m. stays dead until a human notices. Airflow gives you per-task
retries,retry_delay, exponential backoff, and on-failure callbacks that page you. - Reprocessing history. When you fix a transformation bug, you need to rerun the last 90 days, not just tomorrow.
airflow dags backfillre-executes every scheduled interval in a date range through the same code path as normal runs. - Waiting on external systems. Half of orchestration is waiting — for a file in S3, a partition in the warehouse, a third-party API. Sensors handle the polling, and deferrable operators release the worker slot entirely while waiting, so a thousand blocked tasks don't pin a thousand worker processes.
- Visibility. The grid view shows every run of every task across time with logs one click away. When the overnight pipeline fails, you find the failing task and its stack trace in a minute instead of grepping server logs.
Getting Started
Airflow is a service, not just a library, but local setup is two commands. Always install with the constraints file — it pins the exact dependency set the release was tested against:
AIRFLOW_VERSION=3.0.2
PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
pip install "apache-airflow==${AIRFLOW_VERSION}" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
airflow standalone
airflow standalone initializes a local metadata database, starts the scheduler and web UI, and prints admin credentials to the terminal. Put a DAG file in ~/airflow/dags/ and it appears in the UI within about thirty seconds:
from datetime import datetime
from airflow.decorators import dag, task
@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def hello_pipeline():
@task
def extract():
return {"rows": 42}
@task
def load(payload):
print(f"loaded {payload['rows']} rows")
load(extract())
hello_pipeline()
Sharp edges you'll hit in the first hour:
- Bare
pip install apache-airflowproduces a broken install. pip resolves dependency versions that were never tested together and the scheduler crashes on import errors. Always pass the constraints URL shown above. - An old
start_datewith catchup enabled floods the scheduler. A daily DAG dated two years back queues 700+ historical runs the moment you enable it. Setcatchup=Falseunless you are deliberately backfilling. - Top-level DAG code executes constantly. Airflow re-parses every file in the DAGs folder on a loop, roughly every 30 seconds by default — an API call or database query at module scope runs on every parse, not once per run. Keep all real work inside
@taskfunctions. - The quick-start database is not for real use.
airflow standaloneruns on SQLite, which restricts task parallelism and is unsupported for production. Switch the metadata database to PostgreSQL before you evaluate performance.
Features and Use Cases
Pipelines as code. DAGs are plain Python, so you can generate them from config: one DAG factory looping over a YAML list of source tables beats fifty hand-copied DAG files. Code review, version control, and unit testing all apply, because the pipeline definition is just code.
Dynamic task mapping. .expand() creates one task instance per element of a runtime list — you process every file that landed today without knowing the count when you wrote the DAG. Each mapped instance retries independently, so one bad file doesn't force reprocessing the other four hundred.
The provider ecosystem. Hundreds of provider packages ship maintained operators, sensors, and hooks for AWS, Google Cloud, Snowflake, Databricks, and most other systems you'd connect to. Install apache-airflow-providers-amazon and you get S3, Redshift, Glue, and EMR primitives without writing client plumbing.
Data-aware scheduling. Instead of guessing that upstream data lands by 4 a.m. and scheduling for 5, a DAG can trigger when the upstream dataset actually updates. Schedule-guessing is the root cause of most silent staleness bugs; data-aware triggers remove the guessing.
Deferrable operators. Long waits — a sensor watching for a file, a poller tracking a remote job — hand off to the triggerer's async event loop and release their worker slot. The difference is concrete: hundreds of concurrent waits cost a few coroutines instead of hundreds of blocked worker processes.
Realistic uses: nightly ELT into a warehouse, orchestrating dbt model runs, scheduled ML training and batch scoring, report generation, and infrastructure housekeeping like partition cleanup and snapshot rotation.
Ecosystem and Dependencies
Airflow needs a metadata database — PostgreSQL is the safe choice — and an executor that matches your scale. The CeleryExecutor distributes tasks to a worker fleet via Celery, with Redis or RabbitMQ as the broker. The KubernetesExecutor launches one pod per task on Kubernetes, which isolates dependencies per task and scales worker capacity to zero between runs. An official Helm chart handles production deployment on Kubernetes.
For dbt users, Cosmos renders a dbt-core project into one Airflow task per model, so you get model-level retries and visibility instead of a single opaque dbt run task.
Managed runtimes exist on every major cloud — Amazon MWAA, Google Cloud Composer, and Astronomer host the scheduler and workers for you. The trade is real: less operational load, slower availability of new Airflow versions, and less control over the execution environment.
Architectural Overview
The moving parts:
- Metadata database — the source of truth for DAG runs, task states, connections, and variables. Everything else is replaceable; this is not.
- Scheduler — evaluates DAGs against schedules and task states, then queues work. You can run multiple schedulers concurrently for availability and throughput.
- DAG processor — parses DAG files and serializes them into the database, so the scheduler and UI read serialized representations instead of executing your Python.
- Executor and workers — the executor decides where tasks run: in-process with LocalExecutor, on a Celery fleet, or as per-task Kubernetes pods.
- Triggerer — an async event loop that holds deferred operators' waits without consuming worker slots.
- Webserver and UI — the operational console for runs, logs, retries, and manual triggers.
Two design choices matter most in practice. First, the executor abstraction means the same DAG code runs on a laptop with LocalExecutor and in production on Kubernetes — your scaling decision is configuration, not a rewrite. Second, everything flows through the metadata database, which makes state durable and auditable but also makes that database the component to size and monitor first. An undersized metadata database shows up as mysterious scheduler slowness long before anything throws an error.
Pros and Cons
Pros:
- Apache governance and a very large contributor base — the project will outlive your stack decisions, and hiring engineers who already know it is easy.
- The deepest integration catalog of any orchestrator; provider packages eliminate most connector plumbing.
- Python-native authoring with dynamic task mapping handles config-driven pipeline generation cleanly.
- Executor flexibility from a single box to a Celery fleet to per-task Kubernetes pods without changing DAG code.
- Mature operational features: retries, backfills, alerting callbacks, role-based access control, and audit-friendly run history.
Cons:
- Operationally heavy: scheduler, DAG processor, triggerer, workers, webserver, and a database to keep healthy, even for small deployments.
- Batch-scale latency: seconds of overhead between task handoffs makes it the wrong tool for low-latency or event-driven application logic.
- One shared Python environment per deployment unless you isolate tasks with Kubernetes or virtualenv operators — two DAGs needing conflicting library versions is a recurring headache.
- Local testing needs more scaffolding than plain-Python alternatives, because tasks depend on context Airflow injects at runtime.
- Major-version upgrades are real migrations: budget for database migrations and breaking changes when crossing from 2.x to 3.x.
Comparison and Alternatives
GitHub stars
Weekly star gains
Full metrics details
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
Airflow
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
180 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
133 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
179 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
175 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub GraphQL current stargazer cohort, reconstructed from starredAt events and totalCount retrieved in one stable pagination walk. GitHub does not expose historical unstar events, so this is not an exact ledger of past star counts.
159 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Issues opened
Weekly total
Full metrics details
GitHub API observation. Historical values render only when the source provides a real observation for that day.
Airflow
GitHub API observation. Historical values render only when the source provides a real observation for that day.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub API observation. Historical values render only when the source provides a real observation for that day.
101 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Issues closed
Weekly total
Full metrics details
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
Airflow
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
101 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Pull requests opened
Weekly total
Full metrics details
GitHub API observation. Historical values render only when the source provides a real observation for that day.
Airflow
GitHub API observation. Historical values render only when the source provides a real observation for that day.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub API observation. Historical values render only when the source provides a real observation for that day.
101 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Pull requests closed
Weekly total
Full metrics details
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
Airflow
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
101 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Pull requests merged
Weekly total
Full metrics details
GitHub API observation. Historical values render only when the source provides a real observation for that day.
Airflow
GitHub API observation. Historical values render only when the source provides a real observation for that day.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub API observation. Historical values render only when the source provides a real observation for that day.
101 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub API observation. Historical values render only when the source provides a real observation for that day.
142 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Open/closed pull request ratio
Weekly average
Full metrics details
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
Airflow
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
180 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
133 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
179 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
175 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub Search pull-request totals queried for exact UTC days; rolling values are sums of proven daily counts.
159 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Open/closed issues ratio
Weekly average
Full metrics details
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
Airflow
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
180 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
133 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
179 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
175 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub Search issue totals queried for exact UTC days; rolling values are sums of proven daily counts.
159 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Releases
Weekly total
Full metrics details
GitHub release published_at events bucketed by UTC day; drafts are excluded.
Airflow
GitHub release published_at events bucketed by UTC day; drafts are excluded.
180 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Prefect
GitHub release published_at events bucketed by UTC day; drafts are excluded.
133 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Dagster
GitHub release published_at events bucketed by UTC day; drafts are excluded.
178 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Luigi
GitHub release published_at events bucketed by UTC day; drafts are excluded.
179 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Argo Workflows
GitHub release published_at events bucketed by UTC day; drafts are excluded.
175 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Temporal
GitHub release published_at events bucketed by UTC day; drafts are excluded.
159 observed daily rows. Missing days are not fabricated.
| Date | Value |
|---|
Hotness score
Weekly average
Full metrics details
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
Airflow
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
180 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Prefect
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
133 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Dagster
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
178 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Luigi
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
179 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Argo Workflows
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
175 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Temporal
Derived only from GitHub GraphQL starredAt events after daily star totals are reconciled.
159 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.40 × Hot today + 0.40 × Hot week + 0.20 × Breakout.
- Same-day multiplier = stars 1d ÷ max(1, stars 7d ÷ 7)
- Weekly multiplier = stars 7d ÷ max(1, stars 30d ÷ 30 × 7)
- Fortnight multiplier = stars 14d ÷ max(1, stars 90d ÷ 90 × 14)
- Hot today = clamp(70 × log-scale(stars 1d, 1000) + 30 × breakout-scale(same-day, 4))
- Hot week = clamp(75 × log-scale(stars 7d, 5000) + 25 × breakout-scale(weekly, 3))
- Breakout = clamp(35 × breakout-scale(same-day, 4) + 40 × breakout-scale(weekly, 3) + 25 × breakout-scale(fortnight, 2.5))
| Date | Stars | Stars 1d | Stars 7d | Stars 14d | Stars 30d | Stars 90d | Same-day multiplier | Weekly multiplier | Fortnight multiplier | Hot today | Hot week | Breakout | Stored score |
|---|
Reliability score
Weekly average
Full metrics details
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
Airflow
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
180 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
Prefect
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
133 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
Dagster
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
178 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
Luigi
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
179 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
Argo Workflows
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
175 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
Temporal
Derived from the displayed continuity, closure, shipping, liveness, and support-burden components.
159 observed daily rows. Missing days are not fabricated.
Per-day formula: 0.30 × Continuity + 0.30 × Closure + 0.20 × Shipping + 0.10 × Liveness + 0.10 × Support burden.
- Continuity = 35% active ratio 90d + 20% active ratio 30d + 15% PR efficiency 90d + 10% PR efficiency 30d + 10% liveness + 10% observed-history coverage
- Closure = 55% PR merge efficiency 90d + 45% issue close efficiency 90d
- Shipping = 100 × (20% × release ratio 30d + 30% × release ratio 90d + 50% × release ratio 180d); ratios are releases ÷ 2, 6, and 12, capped at 1
- Support burden = 100 − 2 × open issues per 1,000 stars
- Liveness = 100 × exp(−pushed days ago ÷ 120)
Stars, issues, pull requests, and releases are daily observations. Pushed-days and license are repository snapshot-derived inputs and are not independent historical GitHub events.
| Date | Issues opened | Issues closed | PRs opened | PRs merged | Releases | Stars | Open issues | Pushed days ago | Continuity | Closure | Shipping | Liveness | Support burden | License | Observed days | Missing days | Needs healing | Stored score |
|---|
The orchestration field is crowded, and several alternatives are genuinely good.
Prefect is Python-native like Airflow but much lighter to start: decorate functions, run them, and the same code works locally and deployed. It favors dynamic, runtime-determined flows over Airflow's parse-then-schedule model.
Dagster reframes orchestration around data assets rather than tasks — you declare the tables and files your pipeline produces, and lineage, freshness checks, and selective rematerialization come with that declaration. Its local development and testing ergonomics are the best in this group.
Luigi is the elder option: a lean, target-based dependency resolver with no always-on scheduler. It still works for simple batch chains, but development activity has slowed substantially, and I wouldn't start a new project on it.
Argo Workflows is Kubernetes-native: workflows are custom resources, every step is a container, and YAML replaces Python. If your platform team lives in Kubernetes and your steps are polyglot containers, it's a strong fit; if your team thinks in Python, the YAML gets old fast.
Temporal solves a different problem — durable execution for long-running application workflows like payment sagas and order fulfillment. Choose it for reliability logic inside services, not for scheduled data pipelines.
Kestra is a newer declarative orchestrator: YAML-defined flows, event triggers, and a polished UI aimed at teams that don't want pipeline logic in a general-purpose language.
| Project | Repo | Hotness (~6mo) | PRs merged (~6mo) | Star growth (~6mo) |
|---|---|---|---|---|
| Airflow | https://github.com/apache/airflow | 57.93 | 1859 | 527 |
| Prefect | https://github.com/PrefectHQ/prefect | 79.09 | 1568 | 1389 |
| Dagster | https://github.com/dagster-io/dagster | metrics pending | 0 | 300 |
| Luigi | https://github.com/spotify/luigi | metrics pending | 0 | 3 |
| Argo Workflows | https://github.com/argoproj/argo-workflows | metrics pending | 0 | 77 |
| Temporal | https://github.com/temporalio/temporal | metrics pending | 0 | 679 |
| Kestra | https://github.com/kestra-io/kestra | metrics pending | 0 | 36 |
If you want one name: pick Dagster as the best overall alternative. It is the closest feature-for-feature replacement for batch data pipelines, with stronger local testing, built-in asset lineage, heavy ongoing maintenance, and documentation quality that matches Airflow's.
Conclusion
If you're a data engineer running scheduled batch pipelines against a warehouse, Spark, or dbt, use Airflow — its provider catalog and battle-tested scheduler mean you write business logic instead of integration plumbing. Start with the official documentation and keep compute inside your data systems, using Airflow purely as the control plane. A decade of production hardening shows up in exactly the failure paths that bite orchestrators at 3 a.m.
Do not install with a bare pip install apache-airflow — pip resolves provider and core versions that were never tested together, and the scheduler crashes on import errors. The fix is mechanical: always pass the version-matched constraints file URL to pip. Do this first: run the constraints-pinned install shown above, then airflow standalone, and get your first DAG visible in the UI inside ten minutes.