The Airflow DAGs that run paiddaily.io

paiddaily.io is a trading dashboard for DeFi yield markets — static Next.js frontend, FastAPI backend that only reads from Postgres, and 21 Airflow DAGs that keep the data fresh. Here's the full architecture, from chain to screen.

The architecture

The rule: Airflow writes, the API reads. No RPC calls on the request path, no HTTP fetches inline, no background loops inside the API process. Data freshness is a DAG concern. Request latency is an API concern. They never compete.

Aerodrome — 6 DAGs

Aerodrome is the LP management surface. Six DAGs keep pool data, position state, and voting alerts fresh.

aero_positions_and_alerts runs every 2 minutes. Three tasks in one DAG: Multicall3 batch reads for on-chain pool state (tick, sqrtPrice, liquidity), then two parallel branches — range monitor (detects in/out-of-range position transitions and fires push notifications) and wallet sync (Sugar position refresh for every active wallet).

aero_claims_locks_sync runs every 5 minutes. Three parallel read-driven syncs: pool address resolution (resolves on-chain addresses for pools a user has viewed), AERO claim sync (walks claim history for accessed wallets), and veAERO lock sync (refreshes voting escrow state). All three are event-driven — they only process entities that have been accessed since the last sync.

aero_pool_universe_refresh runs hourly. Pulls the full Aerodrome pool universe from DefiLlama — TVL, volume, APR, IL risk for ~425 pools.

aero_v1_positions_sync runs every minute. The cheapest sync — one LpSugar eth_call per wallet. Keeps v1 vAMM/sAMM position data fresh.

aero_vote_market_scan runs every 15 minutes. Walks alive gauges and computes $/veAERO for the current epoch. Feeds the vote market scanner.

aero_vote_reminder runs every 30 minutes. Checks whether lock holders have voted this epoch. If it's Wednesday and they haven't, queues a push notification before the Thursday epoch flip.

Boros — 4 DAGs

Boros covers funding-rate yield markets on Binance and Hyperliquid.

boros_snapshot_and_arb runs every 5 minutes. Fetches market snapshots from the Boros API, computes catalyst-aware forecasts, writes to bronze archive, upserts snapshots, then runs the cross-venue arb recompute — pairing markets by underlying across venues and computing spread, fees, and edge.

boros_market_discovery runs hourly. Market discovery (fetches /markets, writes bronze, upserts new markets) followed by vault refresh (TVL, APR, collateral breakdown).

boros_daily_report runs daily at 00:30 UTC (7:30 PM CST). Historical 7-day MA recompute from the /historical endpoint — refreshes the 7-day moving averages for every active Boros market.

boros_recommendation runs daily at 00:50 UTC (7:50 PM CST). Loads the latest regime data and cross-venue arb opportunities, ranks by edge magnitude × log(open interest), and caches the result for the API.

Pendle — 8 DAGs

Pendle covers yield tokenization markets on Ethereum and Arbitrum.

pendle_price_refresh runs every 5 minutes. Fetches asset prices from Pendle's V3 API and updates the price columns that drive regime classification.

pendle_sync_markets runs hourly. Pages the Pendle V2 /markets/all endpoint for every tracked chain, writes to bronze, upserts via stored proc. Handles new market discovery and relisting.

pendle_new_pool_detection runs every 15 minutes. Drains pendle.market.listed outbox events from the domain event table, looks up market metadata, and upserts ticker legs (underlying, PT, YT, SY) — closing the latency gap between a Pendle market sync and the next hourly ticker harvest.

call_of_the_day runs daily at 12:30 UTC (7:30 AM CST). Gathers candidates from Pendle and Boros regime views, scores them with a multi-factor model, records the winner and runners-up. Powers the hero card on the Pendle today page.

pendle_daily_snapshot runs daily at 00:15 UTC (7:15 PM CST). First retires matured markets (marks them delisted so they don't get a final snapshot), then runs the full classifier — fetches V3 historical data for every live market, computes implied APY, underlying 7-day MA, LP APR, and catalyst-aware forecast. Writes bronze, upserts via stored proc, and detects regime flips.

pendle_daily_analysis runs daily at 00:45 UTC (7:45 PM CST). APY-drift catalyst suggester — detects markets where the 7-day MA shifted more than 150 bps, writes the suggestions to bronze and Postgres, then hands off to the recommendation engine.

pendle_recommendation runs daily at 00:45 UTC (7:45 PM CST). Standalone recommendation engine — loads regime data, ranks opportunities by edge magnitude × log(TVL) within risk tiers, generates action/why sentences, and caches the result for the API.

pendle_morpho_refresh runs daily at 01:15 UTC (8:15 PM CST). Queries Morpho Blue's GraphQL API for markets where Pendle PT tokens serve as collateral. Upserts borrow rate, LTV, leverage caps, and leveraged yield into the Morpho params table — powering the "leverage this PT" cards on the Pendle detail pages.

Tickers — 3 DAGs

ticker_harvest runs hourly. Discovers new tickers from pool underlyings and market references across Aerodrome and Pendle.

ticker_enrich runs hourly at :05. Enriches discovered tickers with price, metadata, and chain data.

refresh_tickers runs daily at 04:00 UTC (11:00 PM CST). Full ticker refresh — reconciles the registry against all known sources.

The shape

21 DAGs. Four domains. Every write goes through a stored proc. Every read comes from Postgres. The shortest interval is 1 minute (aero_v1_positions_sync). The longest is daily (snapshots, recommendations, ticker refresh). The API has no idea any of this is happening — it just reads tables that happen to be fresh.