Part eight · field guide
Case files.
Seven real incidents with the agent itself, each written the same way. Every fact comes from its journal or its build record. The wider list of quiet failures is in Everything that went wrong.
The 422 that rewrote R1
17 July 2026, the first armed drill
What happened. I armed R1 at 12:46 AM and took an access point offline on purpose. R1 asked the controller to restart it. The controller answered HTTP 422: it won't restart a device that's offline.
What it did. It wrote a bare error line, with no verdict and no page. After a re-arm the next two outages went the other way: R1 power-cycled each access point's switch port, and both were seen back online, 78 and 93 seconds later.
What I learned. R1 fires on “access point offline”, so a restart can never fix the thing that triggered it. Power-cycling the port isn't a fallback. It's the only path. The receipts also showed the agent hadn't recorded the uptime it later checked against.
What changed. R1 restarts only an access point that's still reachable. Otherwise it power-cycles the mapped port, and with no mapped port it escalates. Any failed call now becomes an escalation and a page. Still open: for an offline access point, “back online” is accepted without comparing uptime.
Lesson: test the fix against the exact state that triggers it.
The debounce latch
Streaks opened 15–16 July 2026; found and fixed 26 July
What happened. Debounce counts readings before it believes an outage. Nothing ever reset that count, so it became a lifetime tally and both of its tests stayed permanently true. When I found it, all three access points were sitting at a count of 4 with R1 armed.
What it did. On 17 July both R1 actions fired on the very first offline reading of their outage, passing a guard that was already broken. On 26 July a camera went straight to “would act” on its first reading, off a count left over from four days earlier. R2 was switched off, so nothing ran.
What I learned. The R1 drill passed on a broken guard. A counter that never resets proves nothing.
What changed. The count now clears when a device comes back, and a gap longer than twice the waiting window starts a fresh count. “Confirms” means consecutive readings. Ten regression tests fail on the old code and pass on the new.
Lesson: a counter that never resets is not a guard.
67 hours blind
30 July to 2 August 2026, and again 7 to 11 September
What happened. In July the controller software was down for about 67 hours while the internet kept working. The agent logged over 4,000 refused connections and nothing else. In September, after a key change, the agent's own services stopped and stayed stopped for four days.
What it did. In July, nothing. Every failed poll still refreshed its “last seen” time, so its own staleness alarm never fired, and an empty journal looked like a quiet network. In September the publisher kept reporting healthy, because “healthy” only meant “my upload worked”. On restart, the agent worked through 7 queued events up to 85 hours old as if they were new.
What I learned. A silent agent and a quiet network look identical unless something checks the difference.
What changed. A failed poll no longer counts as contact, and a page follows about 4 minutes after contact is lost. Events older than 10 minutes are thrown away with a receipt. The publisher now reports the agent's health, not its own upload, and the dashboard never says “nothing needs you” while the agent is quiet.
Lesson: silence is not health; a failure must never refresh “last seen”.
The breaker tripped on purpose
12 September 2026, the camera drill
What happened. To prove the circuit breaker, the drill used camera power-cycles, which cost about 3 seconds of picture each, instead of access point restarts. The breaker limit is 3 actions an hour.
What it did. Camera-1 fixed and verified, Camera-2 fixed and verified, then a third attempt on Camera-2 was refused by the controller because it was still booting. Two seconds later the breaker opened, every runbook switched off and a page went out. The next two events were held, and I re-armed it a minute later.
What I learned. The breaker works and survives restarts. But the cooldown limited how many fixes a device got, not how close together they were, and a refused attempt still spent breaker budget.
What changed. Each device now has a minimum gap between actions (6 minutes for access points, 3 for cameras), proven the same night. The breaker records what each action achieved, so a trip notice says what spent the budget.
Lesson: prove a brake by tripping it, and read what spent the budget before re-arming.
The never-touch list that guarded an empty port
12 September 2026, revised again 6 October
What happened. The main switch's never-touch list protected a port still labelled as the storage box's. That port was empty, with its power off. Meanwhile the storage box's live port, the downlink to a smaller switch, and the port for a desk device the agent must never touch were all unprotected.
What it did. Nothing touched those ports. The runbooks only ever cycle their own mapped ports. But one mistake in a port map would have been enough.
What I learned. A protection list can guard a label instead of a device. Only one of the exposed ports actually had power enabled, so it was the real hazard.
What changed. The lists were rebuilt from the live port table. On 6 October the agent's own computer moved onto a port of that switch, and that port was added too.
Lesson: check protection lists against the live port table, and again whenever something moves.
Outages it could never act on
26 July to October 2026
What happened. Between the debounce fix and the October rework there were 29 real access point and camera outages outside planned work. 12 lasted two minutes or more; the longest was 122 seconds. Every one stopped at debounce.
What it did. Nothing, and not because it was switched off. The poller then reported a device only when its state changed, so an outage produced one reading. Debounce needed 2 readings, or 75 seconds measured between readings, and the second reading never came. A switched-off agent that got past debounce would have logged “would act”. None did.
What I learned. My build notes blamed the inaction on R1 being switched off. The code said otherwise: debounce runs before the arm check. At the time, all 9 actions in the agent's life had happened during drills. 25 of the outages hit several devices in the same minute, which points at a switch rather than the devices.
What changed. The October rework added the guards first, then had the poller keep reporting a device on every poll while it stays down. Acting now needs 5 polls and 300 seconds. Every real outage in the journal was replayed through the new logic: all 34 came back by themselves within 122 seconds, so none would have been acted on. That's the right answer. If those blips had hung for ten minutes, 23 of them would have been escalated as upstream faults rather than power-cycled on a dead switch.
Lesson: find the stage where it stopped before you accept a reason for why. And fix the guards before the trigger: a runbook that finally fires is only as safe as what it checks first.
The drill that could only fail
27 October 2026, the physical drill
What happened. With the agent armed, I unplugged an access point and then a camera at the patch panel. Each was seen down on every poll, and at poll 6, about 5 minutes in, the runbook power-cycled its switch port.
What it did. Both times the controller answered HTTP 422 and refused the cycle, and the agent escalated. The breaker counted 2 failed actions and stayed closed. The camera's receipt called the refusal “verify failed”, although nothing had been tried to verify.
What I learned. The controller won't power-cycle a port with nothing drawing power. That's a hypothesis, consistent with the September drill's refused second cycle. So an unplugged-cable drill proves detect, guards, act and escalate, but never a successful cycle. The successful cycles on record are on powered ports.
What changed. Before any power-cycle the agent now reads the port. With no link and no power draw it escalates “check the cable” without trying, and spends no breaker budget. A refused cycle is recorded as failed for both access points and cameras.
Lesson: know what a drill can and cannot prove before you run it, and make a receipt say what actually happened.