Thursday, July 23, 2026

An unattended job should never depend on a human's session being awake.

A public page frozen for three days traced back to a publish that had been failing since a login expired, with nothing anywhere reporting it. Fixing the page took an hour. Eliminating the whole category took the day — a credential that doesn't expire, and a dead-man's switch that watches outcomes instead of causes.

Shipped

• Three days frozen, silently — the public stream on sagedaily.io had stopped updating because the automated publish depended on an interactive login session that had expired, and nothing raised a hand when it failed. The page now refreshes itself in the reader's browser while still serving a pre-built version first, so freshness no longer rides on a publish succeeding
• Permalinks recover themselves — a link to an item that wasn't in the last build used to be a dead end; it now goes and fetches the item live instead of showing a not-found page
• No human sessions in unattended jobs — created a narrowly scoped, non-expiring deploy identity that can write to the site storage and refresh the edge cache and nothing else, then moved every site publish across to it. The expiry class of failure is gone, not patched
• A dead-man's switch across every project — the real gap wasn't a credential, it was that unattended work could fail silently for days. Built a shared heartbeat: each scheduled job records its last success, a registry names all of them, and a watchdog raises anything that's gone quiet past its own window. Cause-agnostic by design — it doesn't care why a job stopped, only that it did
• Wired the heartbeat into everything running — audited scheduled work across four projects and connected the ones that matter, so the watchdog reports on real jobs instead of an empty registry
• Credentials that fail loudly — one service was re-issuing a token on every run and never saving it, and a publish token failed quietly when missing; both now persist properly, warn before expiry, and fail with a real error rather than a shrug
• eightballsports.com went live — put the feed on the public internet as a build-time snapshot on static hosting, which costs nothing to run and can't wake up broken. Untangled a domain alias still held by an old account, moved it across, and verified the live site end to end
• The publish became one command — baked the deploy targets into the skill so shipping that site is a single step instead of a remembered sequence
• An audit finding is not an incident — a flagged credential-rotation bug looked like a live exposure until the check that mattered: the service it belongs to has empty credentials and isn't running. Latent, filed, not fixed

Notes

The three-day freeze had a satisfying immediate cause — an expired login — and swapping the credential would have felt like the fix. It wasn't. The credential was one way for an unattended job to stop; the actual defect was that any way at all would have produced the same result, which is nothing visible, for as long as nobody happened to look at the page. Chasing causes one at a time is an endless game, because the next outage always arrives through the one door nobody thought to board up. Watching outcomes is finite: every scheduled job records when it last succeeded, and a watchdog raises anything that's gone quiet longer than it should. That check doesn't care whether the reason was an expired session, a dead network, a bad deploy, or something nobody has thought of yet. Pairing it with a deploy identity that can't expire in the first place closed both halves — one removes a specific failure, the other guarantees the next one gets noticed.