gpg: Signature made Wed Apr 28 20:29:55 2021 UTC
gpg: using DSA key 2099B64CBF15490B
gpg: Good signature from "Andy Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andrew James Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andy Smith (UKUUG) <andy.smith@ukuug.org>" [unknown]
gpg: aka "Andy Smith (BitFolk Ltd.) <andy@bitfolk.com>" [unknown]
gpg: aka "Andy Smith (Linux User Groups UK) <andy@lug.org.uk>" [unknown]
gpg: aka "Andy Smith (Cernio Technology Cooperative) <andy.smith@cernio.com>" [unknown]
Hi,
On Tue, Apr 27, 2021 at 09:39:25PM +0000, Andy Smith wrote:
> A reminder that the greater part of the scheduled maintenance is
> happening shortly at 23:00 BST (22:00Z):
The work last night mostly went well. The timeline was something
like:
22:04Z Customers on "jack" start to have their guests shut
down/suspended
22:17Z Customers on "jack" start to have their guests
booted/restored again
22:20Z Shut down / suspend for "paradox"
22:40Z Boot / restore for "paradox"
22:40Z Shut down / suspend for "hobgoblin"
22:57Z Boot / restore for "hobgoblin"
22:58Z Shut down / suspend for "macallan"
23:38Z Boot / restore for "macallan"
23:55Z Shut down / suspend for "clockwork"
00:12Z Boot / restore for "clockwork"
So, on the whole about 15 to 20 minutes per machine.
Things went wrong with "macallan" in that grub failed to run at
boot, leading to some amount of panic and an extra 20 minutes or so
of downtime. The problem was this:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984426
It was solved by:
1. PXE boot in to Debian installer
2. Choose "Rescue" mode
3. Go through the setup process then select the "Reinstall Grub
Bootloader" option
4. Reboot as normal
I don't yet know why it affected only that server.
There was a 4 hour window for the maintenance but it was all done
after about 2h15m (~00:15Z). Thanks for your patience!
Another round of similar work is planned for on or about 27 May. I'm
afraid we haven't managed to verify that date yet, so it might end
up being slightly different but not before then. That work will
involve the remaining four servers: "elephant", "limoncello",
"snaps" and "talisker".
More formal notifications will go out when the date is confirmed.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
