← Contents

Part four

Secrets, and moving it off my laptop.

Two things that seem boring and then eat a weekend.

Where passwords live

On a Mac, the keychain. On a little Linux box with no screen, a file only that user can read. Don't try to put a desktop keychain on a headless box. It wants a login session that isn't there.

Write the fallback into the code instead — try the keychain, and if that doesn't work, read an environment variable. Then the service file hands it in.

One thing that got me: check what variable name your code actually builds. Mine glued a prefix onto a name that already had the same prefix, so the real name was doubled up and looked like a typo. Setting the obvious one gives you an agent that rejects everything, which is a fun afternoon.

Moving a password between machines

Pipe it. Don't paste it into a chat, don't put it on a command line, don't drop it in a file you'll forget about.

Then check it landed by comparing a fingerprint of it on both ends, not by printing it out. Same length and same hash means it worked, and neither side ever showed you the thing.

The rotation script is part of the system

My worst password problem wasn't a leak. It was a script that only updated some of the places.

It would have reported a clean rotation while the agent was locked out

After I moved the agent to a new machine, the rotation script still wrote the old laptop's keychain and restarted services that I had deleted that same day. Then it checked its work by looking at those same dead services.

So it would have said everything worked, while the actual running agent sat there getting rejected every minute.

Keep a list of every place a password is read and every service that has to restart. When something moves machines, go looking for the scripts that WRITE the password, not just the ones that read its output.

Your rotation script should check the new password against the real thing before saving it anywhere, write every single place, shout if it can't reach one, and then check its work on the machine that actually uses it. Checking your laptop's keychain only proves your laptop is fine, which is the exact blind spot.

Two annoying little things

The clipboard only holds one thing. If you copy a password, then paste a command into a terminal, you just destroyed the password you were carrying. Obvious afterwards. I did it three times in a row before I made a tiny script whose name I type, that puts the password back on the clipboard.

Piping into a script that also has a here-doc doesn't work. The here-doc wins and your piped data just vanishes. No error. That one broke something every single run while the script itself was completely fine.

Getting off the laptop

I ran this on my MacBook for two months. In that time it gave me a four day outage nobody noticed, a sleep setting that killed the poller twice, and an address that moved and quietly broke every push alert. All three went away when I moved it to a little always-on box.

Never have two of these running at once. The switchover is a handoff, not an overlap.

Two copies sharing the same files is worse than either one alone. They both drain the same queue, both write the same state, and whoever writes last wins. Quietly.

The order I'd use:

  1. Build it on the new machine, switched off, with its own scratch folder.
  2. Run the whole test suite on that machine.
  3. Prove it can reach the network controller from there.
  4. Prove something else on the network can reach it. From a third computer, not from itself.
  5. Stop the old one completely, and check its poller has actually stopped moving.
  6. Point the new one at the real folder and start it.
  7. Turn it on. Right now, before the next step.
  8. Move the heartbeat. Never run two.
  9. Point the push alerts at the new address.
  10. Set up whatever copies files back for you to look at.

Turning it on at step seven and not step ten matters. Polling is what actually fixes things and it's already running, so you're only unprotected for a few seconds. If you go do the alert settings first you're unprotected for as long as that takes.

Things that caught me on the way

  • Old state you don't need. Check before copying. Mine was over a week stale and copying it would have been worse than starting clean.
  • History you do need. The log of what it's done should move, or your dashboard goes blank and the new machine looks like it's never done anything.
  • Everything that reads the old file. This one bit hardest. Moving the writer left its readers pointing at a file that stopped updating, and the writer succeeds either way so nothing complains. That includes your tests. One of mine kept comparing against the frozen file and failing on logic that was fine.
  • Time zones. Pin the schedule in the timer, not the machine's clock. Leave the machine on UTC so the logs aren't ambiguous.