Finding problems.

The controller can push an alert at me when something breaks. I can also just ask it every minute. Both are useful and they fail very differently.

Push is fast, when it works. When it stops working it does nothing at all, and nothing at all looks the same as a quiet network. Polling is slower, but if it stops, a timestamp stops moving and I can build a check around that.

Build the important part on the one that can prove it's still working.

So polling is what's allowed to trigger a fix. Push is a head start and a second opinion. It's a hard rule in mine — an alert that arrives by push gets written down and classified, and then explicitly refused as a reason to act.

Dwell time. Read this part.

Your controller probably doesn't raise an alert the second something goes offline. Mine waits about four minutes. It took me a while to figure that out.

Which means, and this one cost me real time:

• Rebooting a device can't test your push alerts. A one minute reboot never reaches the threshold. No alert gets raised, nothing gets sent, and what you see is silence. Silence looks exactly like a broken setup.
• To actually test it you need something off for longer than the wait. I unplugged an access point and left it out for ten minutes.
• Push is structurally minutes behind polling. So if you want quick fixes, they come from polling.

Go measure yours and write it down. Unplug something, note the time, wait for the alert. Everything about how you detect problems depends on that number and I couldn't find it documented anywhere.

Alerts have their own weird logic

Two things I ran into that I wouldn't have guessed.

A rule set to alert on "device went offline" also fires when it comes back, because that's the same alert closing. So a rule with only "offline" ticked delivered me a "reconnected".

And an alert left open from an old problem changes what a new one does. One of mine stayed open for almost five days. The next reboot closed that old alert instead of raising a fresh one, so I got nothing.

When a push alert doesn't show up, go look at the controller's own alert history, not just your logs.

Measure from when the thing happened

Proving push actually works

Don't accept "the settings look right". You want to see a success logged, coming from the controller's own address. Two weaker tests will fool you, and both fooled me.

Testing from the same computer the service runs on doesn't cross the network at all. It proves nothing about anything else reaching it. Use a different machine.

And an error with a neat little JSON body in it is usually your app saying no, not the gateway in front of it. Read your own code before you decide the password is wrong.

The header thing that cost me two days

Put the shared secret in a header, never in the web address. Addresses end up in logs in plain text, and in proxy logs, and in browser history.