← Contents

Part five

Watching the watcher.

The agent can't be trusted to tell you it's healthy. Build the outside view first, not last.

Watch for the missing heartbeat

Each job writes a heartbeat when it finishes. Something separate watches for one that didn't arrive.

That works because it flips the usual thing around. A normal uptime check asks "are you answering?" — and a job that has stopped running answers perfectly, because it isn't a website. Three of my scheduled jobs stopped quietly inside two weeks and every uptime check stayed green the whole time. Correctly green. They were watching the wrong thing.

The heartbeat must never be able to break the job it's watching. Every failure prints a line and carries on. A monitor that can take down the thing it monitors is worse than no monitor.

A heartbeat has to be about the agent, not the job sending it

Green every five minutes, for four days, about something dead

A job published a summary every five minutes, posted fine, and beat green the whole time. The agent it was publishing about had been dead for four days. The summary just got emptier and emptier and nothing noticed, because the heartbeat only ever meant "this job ran".

Make the health check say something about the subject. Is the poller's timestamp fresh? Is the queue draining? Has anything been rejected that shouldn't have been? And if it can't tell, that counts as unhealthy, not healthy.

The one that started all of this

Rejecting everything, for two days, all green

The agent turned away every single alert the controller sent, for two days, with an authentication error. Four services up. Poller fresh to the second. Queue empty. Tests passing. Heartbeat green.

All of that was true. A rejected alert never reaches the queue, so the queue stays empty — which looks exactly like a queue being emptied properly. And the poller is a separate loop that never stopped. The rejections existed only as lines in a web server log that nothing ever read.

I found it by accident, reading a log while doing something else entirely. That's not a plan.

Anything that THROWS AWAY input has to leave a mark somewhere a health check looks. Nothing arriving and refusing everything look identical from outside.

Alerts I'll actually read

Mostly this is about sending fewer.

  • Only when something's wrong. A daily "all good" email trains you to filter the whole folder.
  • Group by problem, not by run. Same problem still happening should tell you once, then go quiet, then remind you in a week.
  • Say when it clears. Something that's silent for both broken and fixed is the same as something that's dead.
  • If it repeats, it has to say something new. I had six identical emails about a stopped job sitting unread. All correct. All ignored.
  • If two systems can report the same thing, make the second one quieter, not silent. Mark it as already handled. Don't drop it — you've traded a duplicate for a blind spot.

Reuse whatever already sends your email

Setting up a new sending address means new DNS records, and mail authentication fails quietly and in ways that are hard to see.

Go look at what you already have first. Mine turned out to hold no password at all — it posts to something else that holds the key and decides the address itself. So the new machine needed one extra secret instead of a whole afternoon of DNS.

Read the right output

Small thing, expensive shape. I searched a job's normal output for a warning that gets written to the error output. Found nothing. Which read as "this never happened" and was the exact opposite of true. I was one sentence away from writing it down as a fact.

Before you write down that something didn't happen, check you were looking in the right place for it.