Hi,
In addition to Lester's email below, I have received one email
off-list expressing concern about the amount of Xen security
advisories recently, and asking whether I was considering changing
to a different hypervisor. I thought I'd respond on-list [also Bcc'd
to the person who contacted me off-list].
I am of course not happy with the amount of serious Xen security
flaws that have been uncovered recently. BitFolk turned 10 years old
this year and in the first few years it was unusual to see one
serious XSA per year. This latest set will have been the third set
in three months.
Most of the problems are being discovered by the Xen project
themselves as part of their ongoing development efforts, or by
developers at the handful of very large companies that use it, e.g.
Amazon and Oracle. I do not believe that code quality has gone down
recently; I think it is more the case that they are getting better
at spotting bugs. Most of these bugs affect all versions of Xen, so
have been present for years.
When considering another hypervisor, what that effectively means is
KVM.
There are probably more companies using KVM and it is certainly a
better known brand name in the virtualisation world, though I
suspect in terms of number of bare metal hosts running it, AWS's use
of Xen would put it ahead there.
If I started BitFolk over again then KVM would probably be the first
thing I would look at, but I'm still not entirely sure that I would
go with it.
Although Xen has seen a lot more security advisories than I would
like, especially in the last 2 years, I do appreciate its security
disclosure process. It's what enables BitFolk to be notified of
security bugs at the same time that huge companies like Amazon and
Oracle find out about them (unless they discovered them,
obviously!), 2 weeks before the rest of the Internet.
Another thing to consider is that Xen are disclosing every bug that
could ever possibly have an effect, not just bugs that are
exploitable in Linux.
When comparing the situation against KVM it's hard because the KVM
project doesn't have a security bug disclosure process at all. They
don't send advisories. The only place they show up is in the
changelog of the Linux kernel, and not all security issues make it
there. I'm sure that any that are known to be exploitable in Linux
do, of course.
So basically, I really appreciate there being a comprehensive list
of advisories for Xen¹ and BitFolk being included in a 2 week
pre-disclosure, neither of which we would get with KVM.
It's true though that I haven't had a look at KVM in a long time and
haven't ever had a proper look at it. I will do that in the next few
months just to get a better handle on things.
As regards making Xen patching a less disruptive process, as you're
probably aware I'm already pushing suspend-and-restore (almost all
of BitFolk's infrastructure VMs do it). I am next going to put
concerted effort into investigating live patching:
https://wiki.xenproject.org/wiki/LivePatch
That's basically similar to the various kernel live patching
efforts, but for Xen (which is booted as a kernel, too).
We're currently running 4.8.x and live patching was only a
technology preview in 4.7. I still don't think it is quite ready yet
and don't expect this to become really usable in production until
I've upgraded all hosts to Debian stretch and moved to Xen 4.9.
I would hope by then that most security bugs can be live patched so
as to not require a reboot.
I am sorry for the disruptions these security patchings are creating
— it's certainly no fun for me either! Not only do I have to do the
work but also the vast majority of my personal and professional
hosting runs at BitFolk too.
Cheers,
Andy
¹
http://xenbits.xen.org/xsa/
On Fri, Sep 29, 2017 at 09:39:04PM +0100, Lester Hawksby wrote:
> Blimey, Xen's giving you some right trouble at the moment!
>
> Thanks very much for all the clear explanation as to what's up. Much
> appreciated. (Doesn't affect me at the moment as my VM is idle after I
> changed my original plans and have been very short of time to start again,
> but the clarity is still making me glad I chose you guys).
>
> Best
>
> Lester
--
https://bitfolk.com/ -- No-nonsense VPS hosting
