← Contents

Part two

Keeping it safe.

Read this before you write any code. Adding safety checks to a thing that already works is how you end up with checks that have never once done anything.

Seven gates

Something only happens if all seven say yes.

  1. A master switch. Off means nothing happens, right now, without uninstalling anything.
  2. A switch per fix. Turn off just the camera one.
  3. An age check. If the alert is old, ignore it. After a four day outage mine worked through a backlog of alerts up to three and a half days old, like they had just happened. The only reason it didn't restart two healthy access points was an unrelated bug.
  4. A wait. Don't react to something that's flapping.
  5. A per-device limit. Don't keep poking the same thing.
  6. A minimum gap. Don't act again before the thing could possibly have come back.
  7. A breaker. Too many actions in an hour and the whole thing shuts itself off and tells me.

Which way should it fail

This is the question I'd think about longest, and there's no single answer.

My age check fails open. If it can't work out how old an alert is, it lets it through. Throwing away a real problem is worse than acting on one that's slightly stale.

My "did it work" check fails closed. If it can't prove the fix worked, it escalates to me. Saying you fixed something when you can't show it is worse than bothering me for nothing.

Write the reason next to each one. "Which mistake is worse here" is a question you can answer. "Should this fail open" isn't.

Check that something changed, not that something is true

This was the sneakiest bug I had, and it's the one that actually scared me.

It said it fixed the camera. It did nothing.

The camera fix cut power to the port, then watched until the camera was connected again. But the camera started connected. So the first check passed immediately, and it reported success having done nothing I could observe.

Take a reading before you act. Then require it to move — an uptime that resets, a counter that goes up. "Is it up?" isn't proof of anything.

Related, and it bit me separately: if you're sampling every five seconds you can't see a three second blip. A negative result is only as good as how often you looked. I had a confident wrong answer in my notes for a while because of that.

The breaker is about runaway, not damage

I went back and forth on whether a fix that failed should count against the limit. It does, and here's why.

If it counts too much, the agent shuts itself off early and I get a message. Annoying, fine, recoverable. If it doesn't count, something stuck in a loop of failing calls can hammer away forever and never trip anything. That's the exact runaway the breaker is for.

Also you can't cleanly skip failures anyway. A timeout isn't proof nothing happened. The call might have gone through and the answer got lost. Counting it is the only safe way to treat "I don't know".

Make every check trigger once

A check you've never seen fire is a guess.

Five of mine were dead. Sitting there in the config, mentioned in the code, and not able to trigger no matter what happened. One was looking at a field nothing ever filled in — and if I'd "fixed" that field it would have quietly switched off a whole fix. One was in the wrong part of a config file and got ignored for three days while printing a warning I never read.

Set aside real time to make each one happen and watch it. It's not polish. It's the difference between having safety checks and having a list of them.

Two rules I'd hard-code

Never let it touch what it needs to exist. Write down the ports and devices the agent depends on and put them on a never-touch list. Then read that list again whenever anything moves. Mine was right when I wrote it, then I moved the agent's own computer behind one of those ports. It was safe by luck, not by design.

One action per problem. Not a sequence, not a retry loop. Do the thing, check, escalate if it didn't work. Retrying is for a person who has looked at it.