Sunday, July 19, 2026

A job that fails every cycle and exits successfully will rot a product in silence for eighteen days.

Two pipelines had quietly stopped producing while reporting that everything was fine. Both got traced, fixed, and drained. Between them, a paused project came back out of storage with its entire front end rebuilt, and a games platform went from a prompt to shipped code.

Shipped

• Eighteen days of nothing, reported as success — the creators digest on paiddaily.io had been frozen since the first of the month. The scheduler ran every half hour and exited cleanly every time, while the step that actually reads the videos failed on every cycle because it was pointed at two local models that had been removed from the machine weeks earlier. Work kept queueing; nothing drained
• Made the failure loud — the cycle now exits with an error when the reading step can't reach a model, so the same rot can't happen quietly again. All model slots converged on the one standard local model that's actually installed
• The rest of the chain underneath it — with the outage cleared, four more problems surfaced in sequence: videos without captions were being recorded as failures rather than skipped, one transient network blip permanently poisoned videos that did have captions, one slow video's timeout aborted the entire batch and re-aborted it every ninety minutes, and throughput was capped at four per cycle. All four fixed; the backlog drained to zero and the digest caught up to today
• Videos stuck in limbo, explained — Sage videos were silently never reaching the public page. A daemon started seven minutes before a fix landed was claiming every transcript job and instantly failing it. Reprocessed the stuck ones, killed the stale daemon, and fixed the source: a worker now refuses to claim a kind of job it isn't configured to run
• A health monitor for the pipeline — instead of waiting for the next silent stall, added a check that watches for work sitting too long and raises it, running on a schedule of its own
• Off-topic became a label, not a disappearance — the relevance filter was hiding videos outright, which meant a wrong call was invisible. Everything shows now; off-topic items just carry a "commentary" badge, and the judgement itself is unchanged
• truthers.io rebuilt and repositioned — pulled the project back out of storage and merged two separate front ends into one app, applied a dark follow-the-money design chosen from three directions, made every page full-width and responsive, and simplified the landing around a single claim: being the authority on what government is doing, federal to state to city. Established the missing staging branch and ran the full flow to live
• A games platform, from prompt to shipped — built a game-agnostic contest spine for eightballsports.com and three games on top of it — squares, strip cards, and baseball — with payout windows the host configures, short public addresses, an unattended score poller, and a lobby with filters. Strip cards were modelled wrong first and rebuilt against a real reference
• The money question, answered carefully — free-to-play social organiser is the default and stays the default; optional pools settle non-custodially, with the platform acting only as the outcome resolver, never holding funds and never taking a cut. The escrow contract is written and tested against a test network only; real-money play is gated behind legal review, a location gate, terms and an age check, and that gate is unmet
• The hidden tools link came back — a public nav link had been removed a week earlier while the routes kept working; surfaced it again

Notes

Both frozen pipelines shared one property: they exited successfully. That's the part worth sitting with. A crash gets noticed, because something screams. A job that runs on schedule, does no work, and reports success is indistinguishable from a job with nothing to do — and eighteen days is how long that takes to surface when the only symptom is a page that stopped changing. The durable fix in both cases wasn't the model name or the daemon; it was deciding what counts as failure. A cycle that can't reach a model now exits with an error. A worker that isn't configured for a kind of job now refuses to claim it instead of claiming it and failing instantly. Neither change fixes anything by itself. They just make the next outage arrive as an outage instead of as a slow disappearance.