Part six · the useful one
Everything that went wrong.
All of these actually happened, building this agent and the scheduled jobs around it. I've grouped them by shape rather than by which bit of my setup they hit, because the shapes keep coming back and your gear won't match mine.
Quiet things that look healthy
Rejecting everything is not the same as nothing arriving
Two days of turning away every alert, with an empty queue and a green heartbeat. The empty queue was the problem — nothing arriving and refusing everything look the same from outside.
Anything that throws away input has to leave a mark a health check can see.
A heartbeat about the wrong thing
A job beat green every five minutes for four days while the thing it reported on was dead.
Check facts about the subject, not about the reporter.
A crash loop that never counts as failed
Set something to always restart with a ten second wait and it can never hit the default limit. So it restarts forever, never officially fails, and never triggers your failure handler.
Set the limit window on purpose instead of taking the default.
A skipped step that reports success
A check whose target had moved carried on "passing" while checking nothing at all.
It happened again after the move. A config test looked for a file on my laptop, didn't find it on the new server, printed a note and passed. So on the one machine that mattered it checked nothing, and on the old one it failed on settings that were out of date. Same code, opposite answers.
A missing thing is a failure. Never a skip. And test the rules that should never change, like "this port is protected", not settings you'll tune later.
Safety checks that could never fire
Five of mine. All in the config, all mentioned in the code, none of them able to trigger.
- One matched a field nothing ever filled in. Worse, fixing that field would have quietly switched off a whole fix.
- One was in the wrong section of a config file. Ignored for three days while printing a warning on every start that I never read.
- One checked something that was already true before it acted.
- One protected a port by its label, and the device had moved to a different port.
- One was set up right but could never be reached, because an earlier step always returned first.
A safety check you've never watched trigger is a guess, not a safety check.
Checks that prove nothing
Checking a state instead of a change
Cut power to the port, then watched until the camera was connected. It was already connected. Instant pass, nothing actually verified. My only false success, and the one that bothered me most, because it wasn't broken — it was confidently wrong.
Take a reading first, then require it to move.
Looking less often than the thing you're looking for
A three second blip is invisible if you check every five seconds. I had a confident wrong conclusion written in my notes before I went and checked the physical evidence.
A negative is only as good as how often you looked.
Time
A wait I didn't know about
The controller waits about four minutes before it decides something is offline. So rebooting a device to test push alerts produces nothing, and nothing looks exactly like a broken setup.
Go measure it. Write the number down.
Starting the clock in the wrong place
Declared an alert missing after fourteen quiet minutes. The device had dropped one minute earlier. It showed up 43 seconds later.
Measure from when the thing happened, not from the last line in the log.
A catch-up run at a silly hour
A timer set to make up missed runs fired one straight after a schedule change and mailed a weekend report at half past midnight on a Sunday.
Decide per job whether a late run is useful or just wrong.
Things that were right when written
This is the one I'd most want you to read, because nothing is actually wrong anywhere.
The port list didn't change. The network moved under it.
A list of switch ports the agent must never touch. Correct the day I wrote it. Months later I moved the agent's own computer to sit behind one of those ports.
So cutting power to that port would now cut off the agent in the middle of doing something. The list never became wrong. The thing around it moved.
And searching for the computer's name would never have found that list, because the list doesn't mention it.
When something moves, go read every list that protects it or that it now sits behind. Not just the documents that name it.
A plan that was accurate when it was approved
A build plan describing the wrong operating system, the wrong network, the wrong address and the wrong storage. All true when written, all changed afterwards. Somebody following it builds the wrong thing very confidently.
After a move, read the docs against the running thing. Expect the doc to be the stale one.
Fixed in one place only
A stale comment got corrected on the live machine and not in the source I build from. So the running thing was right and the source of truth was wrong, which is backwards from how you'd assume that goes.
Compare source against live on VALUES, not on text. And expect the source to be the stale one — the live machine is where the pressure to be correct actually lands.
Things that describe themselves
A report that lied about how it was made
A daily report's own footer said it had run in test mode and written test files, leaving the real ones alone. There were no test files anywhere. The output was real. The footer was just a leftover line from an older setup.
The cost wasn't the wrong label. Somebody read that footer and spent real effort working out how test files might be ending up with real names. Both of their theories were sensible. Both were wrong, because the sentence was simply false.
A sentence describing how something was made is evidence about the writer, not about the system. When it disagrees with what you can actually see, the sentence is probably the liar. Say so, and don't invent a mechanism to explain it.
That one has a second half worth keeping. The footer was wrong because a step that used to rewrite it got deleted during a move, and the thing it fixed didn't. If you delete the thing that corrects something, do the correcting.
Passwords
- A rotation script that updated some of the places, restarted services that no longer existed, and then checked its work by looking at those same dead services.
- One password in two places, changed in one. The other thing failed quietly for days.
- A key that only existed in memory. Worked all day. A reboot would have locked everyone out of the always-on box.
- The clipboard holding one thing, and me pasting a command over the password I was carrying. Three times.
Deploys and edits
- A deploy that went green having uploaded nothing, because a filter skipped my change. Match the commit it actually built to the one you meant.
- Editing a rule in a web interface quietly removed its password header. On all five rules. The form looked right before saving. Close it and open it again.
- Opening a rule just to look at it, and saving, wiping its settings.
- Editing a config file a running program holds in memory. It writes its own copy back over yours.
Building it with an AI
Almost all of this got built in conversation with an AI. That worked. It also had a very specific way of being wrong, and it's worth knowing.
Nearly every mistake was a confident no. Not "this works" when it didn't. "This isn't happening" when it was.
No alert arrived. The warning never fired. There's no reservation set. Each one came from real evidence that just wasn't enough — the wrong time window, the wrong output, the wrong machine. They're much harder to catch than a wrong yes, because there's nothing there to look at.
- It read a setting off the client and decided the server didn't have one. That information only exists on the server.
- I handed it a stale list once and got back a well-argued warning about a danger that didn't exist. The reasoning was fine. The input wasn't.
- The running code turned out to be more correct than every document describing it, more than once. Check the code before you trust the write-up.
- Two commands in one go printed overlapping output and we both chased a duplicate that was never there.
What fixed it wasn't better prompting. It was five questions before accepting any "no", which are on the next page, and I run them on myself now as much as on the assistant.