Where we are today
We are pleased to announce that we’ve completed the first round of update reboots as of the evening of Thur Jan 11th. These reboots consisted of updated kernels with Kernel Page Table Isolation (KPTI) and CPU firmware (microcode) updates for a handful of our production systems, namely Intel Haswell, Broadwell, Skylake architectures. Read more
We’ve had an incredibly busy couple of days and wanted to take a few minutes provide an update on where Nexcess is at with Meltdown & Spectre patching.
As is often the case with these kind of situations, the landscape has evolved a bit since our original posting. The most notable of which is that there is an increasing amount of Proof-Of-Concept (POC) code in distribution that demonstrates taking advantage of Meltdown & Spectre vulnerabilities. This raises the threat of the vulnerabilities as quite often these POC’s are used as the basis for creating malicious exploits. At this time however, we have not seen nor have industry peers we work with, any targeted attacks or exploits against these vulnerabilities.
As you may be aware, a number of serious vulnerabilities have been disclosed that affect a wide set of CPU architectures. These vulnerabilities (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754) were disclosed this week by Google’s Project Zero team and other information security professionals. A rapid response strategy is currently under review for emergency maintenance to patch these vulnerabilities, which will require a reboot of all shared, dedicated and cluster systems. Read more
As of 1:45AM EDT on April 8, 2014, all Nexcess managed systems vulnerable to CVE-2014-0160 (Heartbleed) were patched. This security vulnerability is widespread with multiple operating systems globally and not a Nexcess-specific issue.
CentOS released an official OpenSSL patch removing the recently discovered vulnerabilities. OpenSSL was upgraded seamlessly. However, all services linked against the older vulnerable version of OpenSSL had to be restarted to apply the newly patched OpenSSL version. These services include: Apache, PHP-FPM, InterWorx and mail services (imap4-ssl, pop3-ssl, smtp, smpt2) and there was a very brief service interruption as these services were restarted.
The online Heartbleed testers such as http://possible.lv/tools/hb/ simply connect to the server and see if the heartbeat feature is enabled. The CentOS patch installed on our systems (https://bugzilla.redhat.com/attachment.cgi?id=883475) actually fixed the issue vs. simply disabling the heartbeat feature altogether. This means while the system is indeed patched, the online checkers will still show the system as vulnerable as they can’t tell if you are running the patched version or not since there is no known PoC to test against.
If you’ve ever had to hard restart a Linux server, you know when it starts up you’ll see a message about your system being shut down uncleanly and it will do an fsck. But, how does it know you shut it down uncleanly?
The short story is, if the server finds a file at /.autofsck on boot, it knows you didn’t perform a clean shutdown. The /.autofsck file is put there by a startup script and removed when you do a clean shutdown using a commands such as halt, poweroff, shutdown, or reboot. When you perform an unclean shutdown, the shutdown scripts are never run and the /.autofsck file is never deleted, thus an fsck is initiated due to the unclean shutdown. Read more
du is a utility to tell you the disk usage of files or directories. Usually when people run it, they want the output in a human readable format like 314M, 2.7K, and 161G which are a lot easier to read than 314159265, 2718, 161803398874 respectively.
When people are looking for directories using the most storage, they can usually only care about directories using a gigabyte or more, so people will pipe the output of du to grep 'G' to only see them. This works OK but you might miss a directory that is using 987M of storage and the grep will also match a directory that has ‘G’ in it somewhere which an be annoying.
After having done the grep 'G' trick enough times and seen it shortcomings, I figured there had to be a better way to do this. I found two ways instead, one which is incredibly easy and useful and a slightly more complex one. Read more
I think cron is a wonderful creation. It is simple in what it does while being extremely useful. Allow me to present a short introduction to it for those unfamiliar, then I will show you a handy trick you may need someday.
Brief Introduction to Cron
You can set any command to run at any time (or repeatedly at a set interval) by using cron. The name comes from “chronos” – the Greek word for time. If you have Linux, you’re going to have cron. Run the command “crontab” to edit or create the jobs for your user.
The syntax seems a little cryptic at first, but it is very straightforward. You enter a series of numbers and symbols followed by the command you wish to run. The numbers and symbols state the desired time and day to run the command. They are separated by a space as follows: Read more
We run the Nexcess public mirror for distros and software our servers and employees use. Running our own mirror reduces our bandwidth usage since the traffic for updates and installations of our CentOS servers never leaves our network. It also makes doing installs and updates much faster, it’s never fun installing something and your server ends up using a slow mirror on the other side of the country. For our employees, being able to download an ISO for a new distro on a big fat pipe from a local mirror is a nice benefit.
You’d think running a mirror would be relatively simple, just set up an rsync cron job and call it a day, but having run our mirror server for a year I’ve learned and done several things that have made our mirror server operate smoothly:
When you do a fresh install of x86_64 CentOS 5, you might be surprised and annoyed to find yum trying to install 32 bit packages on your 64 bit server. You’ve got a 64 bit processor and operating system so why is it trying to install these un-needed 32 bit packages?
CentOS comes with multilib support, since your 64 bit processor can run 32 bit binaries, yum sees no issues with having 32 bit and 64 bit packages installed at the same time. If you look at a repo for x86_64 you’ll even see a bunch of i686 packages available to the x86_64 release. It seems like a feature most people would never need but a good example of when you’d want to install a 32 bit package on a 64 bit OS is to get something like flash support which only has a 32 bit package. I’ve also seen some RPMs get released exclusively as 32 bit packages.