This is a follow up to our original post.
We’ve had an incredibly busy couple of days and wanted to take a few minutes provide an update on where Nexcess is at with Meltdown & Spectre patching.
As is often the case with these kind of situations, the landscape has evolved a bit since our original posting. The most notable of which is that there is an increasing amount of Proof-Of-Concept (POC) code in distribution that demonstrates taking advantage of Meltdown & Spectre vulnerabilities. This raises the threat of the vulnerabilities as quite often these POC’s are used as the basis for creating malicious exploits. At this time however, we have not seen nor have industry peers we work with, any targeted attacks or exploits against these vulnerabilities.
Further, we currently have a clearer understanding and consensus among industry peers on how to adequately patch for the vulnerabilities. In short, there are two levels of patching required, one is an OS level kernel update and the other is a software update to the physical processor called a microcode update. In order to effectively patch longer term, both must be applied however the criticality of the OS kernel level update can not be understated as it begins to narrow the scope of how these vulnerabilities can be leveraged.
In the last 56 hours, we’ve applied kernel updates and reboots to 86% of internal and customer systems across our global presences through 13 emergency maintenance windows. The remaining systems to patch are those which had conditions that did not allow for immediate patching or, quite simply, that we did not have the scheduling capacity to get to yet.
It is our intention to update and reboot remaining systems over the next 36-48 hours. We will begin customer outreach and emergency maintenance scheduling for these remaining systems by no later than Monday Jan 8th 12:00PM ET.
The microcode processor updates take a different form, traditionally as BIOS updates that can be fraught with risk. However, the Linux kernel provides for a ‘microcode loader’ feature that can be used to, on boot (after BIOS POST), update processor microcodes in a far less risky fashion. This is our preferred approach and that of a larger part of our industry peers. However, these microcode updates even when applied by the Kernel microcode loader, will require a reboot.
We’ve received updated microcode packages for a select few processor types, namely Intel Haswell, Broadwell, Skylake architectures. As such, when applying kernel updates and reboots across our platform, we’ve been updating the microcodes as well. The issue here is these specific processor architectures only make up a small percentage of our production server population, roughly 19%. This means that in the near future, when additional microcode updates are distributed by Intel and OEM vendors such as Dell, we will need to schedule subsequent reboots.
We expect the broader microcode updates and then subsequent reboots to be on a time scale of 1-4 weeks out, potentially longer, depending on release and distribution/testing timeline from Intel and OEM vendors. If at any time an emerging threat warrants it, we can escalate this procedure from kernel microcode updates to BIOS level microcode updates (being mindful BIOS updates take significantly longer across thousands of servers).
When we have more information available and more accurate timelines on microcode updates, we will send out notifications accordingly as we begin to schedule emergency maintenance windows.
In the coming days, weeks and months ahead we fully expect that the landscape related to these vulnerabilities will continue to change. There is an unprecedented focus now on microprocessor architecture and we may well see additional flaws or attack vectors identified. To that end, we’ve been invited to join an industry technology group lead up by Scaleway, Packet, OVH and Linode to coordinate our response, share information and coalesce around best practices. This group has subsequently grown to include DigitalOcean, Vultr, Exoscale, Tata and most recently representatives from RedHat and Amazon.
Last but certainly not least, a very big thank you goes out to everyone here at Nexcess who has been directly or indirectly involved in these patching efforts. It has been an absolute pleasure to see our teams come together to pool resources, keep each other motivated and grind through these updates. This is not an easy time for anyone in the technology space but having some of the best talent in the industry and people that commit themselves without question, make it a little easier to get through the long nights.
We appreciate your understanding and patience as we complete this process. If you have any questions or concerns, please reach out to our Support team via https://portal.nexcess.net.Posted in: General, Linux, Nexcess, Security