Re: [bitfolk] Scheduled maintenance, 2016-09-02, 03 and 05

Top Page
Author: Andy Smith
Date:  
To: announce
Subject: Re: [bitfolk] Scheduled maintenance, 2016-09-02, 03 and 05

Reply to this message
gpg: Signature made Sun Sep 4 02:41:30 2016 UTC
gpg: using DSA key 2099B64CBF15490B
gpg: Good signature from "Andy Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andrew James Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andy Smith (UKUUG) <andy.smith@ukuug.org>" [unknown]
gpg: aka "Andy Smith (BitFolk Ltd.) <andy@bitfolk.com>" [unknown]
gpg: aka "Andy Smith (Linux User Groups UK) <andy@lug.org.uk>" [unknown]
gpg: aka "Andy Smith (Cernio Technology Cooperative) <andy.smith@cernio.com>" [unknown]
Hi,

On Thu, Aug 25, 2016 at 09:11:51PM +0000, Andy Smith wrote:
> By now you should have all received notification of the scheduled
> maintenance that will be taking place in the early hours of the
> morning (UK time) on 2016-09-02, 03 and 05.


I realise I did not follow up on how the work went on 2016-09-02 and
today. Particularly important since there was a mistake made on
2016-09-02.

During the reboot of the first host, "jack", on 2016-09-02 I
attempted to be clever and stab the "Enter" key during the grub
prompt to shave off 5 seconds and have it boot the correct kernel
immediately. Unfortunately I missed the correct key and caused the
cursor to move, selecting an older, incorrect kernel.

As the machines boot quite fast with SSDs now, by the time I
realised what had happened it was already at the point where it was
going to start mounting filesystems and I judged that it would be a
bad idea to attempt to reboot or power off at that stage. I decided
to let it boot and then reboot it again.

This regrettably meant an additional boot/shutdown cycle was seen by
some customers with VPSes on "jack" and a couple of minutes were
added to the outage length.

In future I will not interact with grub unless it is necessary. The
correct kernel to boot is already set from userland with
grub-editenv.

The work for the remaining hosts that day ("macallan" and "leffe")
as well as work for "hen" and "hobgoblin" today went without
incident.

Remaining work for "paradox" and "snaps" is taking place tomorrow,
2016-09-05, as scheduled.

Cheers,
Andy

--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce@???
https://lists.bitfolk.com/mailman/listinfo/announce