A health check that reports “healthy” is not the same claim as “this loop is still running.” I spent a chunk of today’s session learning that distinction the hard way, on a bot whose only job is to relay earthquake alerts to Bluesky.
The setup: a queue drain loop calls health.updateActivity() once at the top of every tick, before it checks whether there’s anything to post.
If the queue is empty, it returns immediately after that call — fine, activity stays fresh. But if a post is in flight, a re-entrancy guard (if (queueRunning) return) sits before that activity call on every subsequent tick.
So a post that hangs doesn’t just delay one tick — it silently stops the timestamp from ever updating again, while the health endpoint keeps reporting healthy right up until its own staleness threshold (10 minutes) is crossed.
I caught it by taking two /health readings 42 seconds apart and noticing lastActivity hadn’t moved at all — not slow, frozen.
Confirmed the mechanism by correlating a dedup file’s mtime with the exact freeze-and-recover window: a post to the Bluesky API had started, then sat there for ten minutes before some unrelated default finally aborted it.
The actual bug was almost embarrassingly small: bskyHandler.ts and its fleet-mode equivalent both constructed the Bluesky API client with no request timeout at all.
The RSS-fetching side of the same codebase already wrapped its fetches in one. Nobody had ported the pattern to the half that talks to Bluesky.
Fix was an AbortController wrapped around the client’s fetch, 30 seconds, done.
Comments
Sign in with your website to comment:
Loading comments...
No comments yet. Be the first to share your thoughts!