As you may be aware, a number of serious vulnerabilities have been disclosed that affect a wide set of CPU architectures. These vulnerabilities (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754) were disclosed this week by Google’s Project Zero team and other information security professionals. A rapid response strategy is currently under review for emergency maintenance to patch these vulnerabilities, which will require a reboot of all shared, dedicated and cluster systems.The vulnerabilities, known as side-channel speculative execution or Meltdown and Spectre, have the potential to allow code to execute on a CPU and access regions of memory that should otherwise be protected from access. This is a vulnerability that has existed for more than 20 years in modern processor architectures like Intel, AMD and ARM across servers, desktops and mobile devices.
Given the seriousness of this set of vulnerabilities, a rapid response is required to ensure our customers are protected. To be clear, there are currently no known exploits circulating that take advantage of these vulnerabilities. However, since details and code fixes are now publicly available, it is only a matter of time before attacks develop around these vulnerabilities.
The immediate security impact to our customers is negligible but has the potential to change. There are currently no known exploits in the wild that are taking advantage of these vulnerabilities. With a rapid patching schedule, it is our goal to ensure customers are protected before any exploits are made available.
The immediately available patches have been in the works for 3 months by various groups and vendors, such as Linux Kernel developers, Microsoft, Intel, Google, and Amazon. These patches represent the best mitigation techniques, known as Kernel Page Table Isolation (KPTI), to ensure code cannot execute to access protected regions of memory.
There have been reports that KPTI patches will impose a performance penalty, as much as 30%+. These reports, while not entirely untrue, they are very workload specific and are not representative of a blanket performance drop. In our own testing, as well as testing by other organizations, the day-to-day performance impact is expected to be negligible, at or around 5%.
The KPTI patches are not expected to impact page load times, database operations or execution of other tasks on our shared, dedicated or clustered platforms. The cases in which more tangible performance impacts can be seen, upwards of 5%, are on systems that are resource-bound (overloaded) and already running at capacity.
Our Systems Operations team will review all existing systems, shared, dedicated and clusters, to ensure none are at risk of suffering more significant performance impacts. If any systems are identified as overloaded, corrective action will be evaluated on a case by case basis and customers contacted as required.
The Nexcess Systems Operations and Data Center Operations teams will begin updates in batches beginning the morning of Friday, January 5th, 2017. As our maintenance response plan proceeds over the next 12-24 hours, customers will be notified of a date and time window in which patch maintenance will be conducted.
The patching procedure that will be executed on a per-system basis, generally, will be as follows:
- Validate availability and date of backups.
- Apply appropriate kernel updates and any dependent packages. No other software will be updated as part of this maintenance.
- Validate that the kernel update applied successfully.
- Perform a graceful reboot of the system.
- Once the system is back online, ensure all services are operating as intended and websites are loading.
This process is expected to result in as much as 15 minutes of downtime, per system. However, the average is likely to be less than this. During this downtime, all websites and services hosted on a system scheduled for maintenance will be inaccessible.
We appreciate your understanding and patience as we complete this process. If you have any questions or concerns, please reach out to our Support team via https://portal.nexcess.net.