gpg: Signature made Sun Sep 10 01:01:59 2017 UTC
gpg: using DSA key 2099B64CBF15490B
gpg: Good signature from "Andy Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andrew James Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andy Smith (UKUUG) <andy.smith@ukuug.org>" [unknown]
gpg: aka "Andy Smith (BitFolk Ltd.) <andy@bitfolk.com>" [unknown]
gpg: aka "Andy Smith (Linux User Groups UK) <andy@lug.org.uk>" [unknown]
gpg: aka "Andy Smith (Cernio Technology Cooperative) <andy.smith@cernio.com>" [unknown]
Hi,
On Tue, Aug 29, 2017 at 05:27:13PM +0000, Andy Smith wrote:
> So, we're going to have to patch everything and reboot before then.
> This will very likely be taking place over three nights starting in
> the early hours (BST) of Sunday 10 September, but we will be sending
> out an individual email to every customer confirming when they will
> be affected.
For the last hour I've been working on "elephant", the first (live)
server to be patched and rebooted.
Unfortunately I made an error, which initially I thought would only
cause a few extra minutes of downtime, but turned out later to
require a complete reboot with not even the ability to cleanly shut
down VPSes.
That has now happened, and all customer VPSes on "elephant" should
now have booted. I am now going through our Nagios to check for any
that might be having problems.
I am really sorry that this happened. I could have tested my
proposed action on test hardware but I was sure I had done it before
without incident, and I was wrong.
I can't at this stage go into exact details regarding what my error
was because this would reveal details about where the security
vulnerabilities lie, so that will have to wait until after the
embargo has ended on Tuesday 12th. I will give more details then.
Apologies again,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
