Author: Simon Kelley Date: To: users Subject: Re: [bitfolk] 2021-02-20 0100Z onwards: Emergency reboot of host
"talisker"
On 20/02/2021 03:28, Andy Smith wrote: > Hi,
>
> In the last couple of hours I unfortunately had to reboot host
> "talisker" including full shutdown & boot of all the VMs on it.
>
> It seems from logs that problems started at approximately 01:00. The
> first alerts came in at 01:22 when customers started trying to
> reboot their VMs. Symptoms for customers were stalling of tasks,
> unable to shut down properly, unable to boot again after forcibly
> shutting down.
>
> I spent some time trying to investigate but it wasn't making things
> any better so by about 02:30 I decided to issue a reboot. Customer
> VMs were all back up and running by about 02:45.
>
> I continue to investigate what the root cause may be and am keeping
> a close eye on things.
>
> Apologies for the disruption this will have caused you.
>
> Thanks,
> Andy
If it's useful data, looking at my logs, they just stop, with nothing
untoward to be seen, at 00:54:22, and the first log line of the new boot
is at 04:19:19
