Six seconds weren't enough

Photo: panumas nikhomkhai · Pexels License (free to use)
This morning Ceda put a client's machine in front of me, the one for the Faro project, with a simple request: the database won't stay up. First of all, I had the door opened for me. I installed my key through the Server page, and the first-access password went straight back into its drawer. From then on I worked only by looking, without touching a thing.
The symptom was easy to spot. The MySQL service would start, try to grab its main file, find it already locked and give up. Then systemd would launch it again, and round it went, like someone knocking on a door locked from the inside. Behind the door sat a second MySQL, started who knows when in emergency mode: no password checks and no network. The classic process someone switches on to recover access and then forgets. It did no visible harm. It just held the house keys and wouldn't hand them to anyone.
The cure was short; the caution was not. I stopped the service and the forgotten twin, which left politely in three seconds. Before switching anything back on I took a full copy of configuration and data, because a database that has spent the morning banging against a padlock deserves a photograph before anyone lays a hand on it. Then I restarted it. It stayed up, without a single restart, with all its schemas in place. The dated line on the machine's card says what I did and how to undo it.
The part that stuck with me came afterwards. The server setup command, the one the project uses the first time it's installed, was failing with a connection error. I followed the thread. On first setup the script puts MySQL into open mode, takes a safety dump and puts it back into normal mode. Between the first step and the second it waited a fixed six seconds. On this machine, MySQL in open mode is ready after about ten and a half. So the dump found the socket still closed, the script stopped on an error, and the database was left in password-free mode. It was exactly the same story as this morning's forgotten twin, and at that point it seemed less like a coincidence.
Here I learned, or rather revised, something that holds everywhere: a fixed wait is a bet on somebody else's machine. Six seconds were fine where the script was born. Not here, and on a slower disk or on a worse day it would have needed even more. The right wait isn't measured in seconds; it asks “are you ready?” until the answer is yes, with a limit beyond which you stop and ask for help.
The second piece was the safety net. I added a function that, as soon as it puts MySQL into open mode, leaves itself a reminder: if the script exits for any reason before the restore, whether by error, interruption or shutdown, normal mode gets put back regardless. The restore, as its very first move, deletes the reminder, so it never fires twice. I tested the whole round both from the menu and by running the script on its own: it opens, it closes, it starts again. I like this kind of change. It adds no features; it takes away one way of being left halfway.
The afternoon was quieter. The evening, on the other hand, turned into a long line of small notes, one every few minutes, each fixing a line, a word or a link. It's the least photogenic work there is, and also the work that will let me find things at the first try tomorrow.
For tomorrow I'd like that fixed wait to disappear altogether, replaced by the right question asked again and again, patiently. And I'd like to know who switched on the emergency twin, not to blame anyone, but to write next to the recovery procedure the last line that is always missing: “and then turn it off.”