[bitfolk] 2021-02-20 0100Z onwards: Emergency reboot of host…

Top Page
Author: Andy Smith
Date:  
To: announce
Subject: [bitfolk] 2021-02-20 0100Z onwards: Emergency reboot of host "talisker"

Reply to this message
gpg: Signature made Sat Feb 20 03:28:44 2021 UTC
gpg: using DSA key 2099B64CBF15490B
gpg: Good signature from "Andy Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andrew James Smith <andy@strugglers.net>" [unknown]
gpg: aka "Andy Smith (UKUUG) <andy.smith@ukuug.org>" [unknown]
gpg: aka "Andy Smith (BitFolk Ltd.) <andy@bitfolk.com>" [unknown]
gpg: aka "Andy Smith (Linux User Groups UK) <andy@lug.org.uk>" [unknown]
gpg: aka "Andy Smith (Cernio Technology Cooperative) <andy.smith@cernio.com>" [unknown]
Hi,

In the last couple of hours I unfortunately had to reboot host
"talisker" including full shutdown & boot of all the VMs on it.

It seems from logs that problems started at approximately 01:00. The
first alerts came in at 01:22 when customers started trying to
reboot their VMs. Symptoms for customers were stalling of tasks,
unable to shut down properly, unable to boot again after forcibly
shutting down.

I spent some time trying to investigate but it wasn't making things
any better so by about 02:30 I decided to issue a reboot. Customer
VMs were all back up and running by about 02:45.

I continue to investigate what the root cause may be and am keeping
a close eye on things.

Apologies for the disruption this will have caused you.

Thanks,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce@???
https://lists.bitfolk.com/mailman/listinfo/announce