We run the Nexcess public mirror for distros and software our servers and employees use. Running our own mirror reduces our bandwidth usage since the traffic for updates and installations of our CentOS servers never leaves our network. It also makes doing installs and updates much faster, it’s never fun installing something and your server ends up using a slow mirror on the other side of the country. For our employees, being able to download an ISO for a new distro on a big fat pipe from a local mirror is a nice benefit.
You’d think running a mirror would be relatively simple, just set up an rsync cron job and call it a day, but having run our mirror server for a year I’ve learned and done several things that have made our mirror server operate smoothly:
- Use a lock file on your cron job – This will prevent your mirror from running two rsyncs at the same time. When there’s a new CentOS release, every mirror is trying to get their local copy from the main CentOS rsync server, it can take a long time to get your copy. Depending on how often your cron job runs, you might end up having two cron jobs running simultaneously, trying to download the same copies of the same stuff. This only adds to the problem the main CentOS rsync server is having.
- Receive and read errors from your cron job – If your mirror server is having problems, you should know about it. I use this in conjunction with the lock file described above to know if rsyncs aren’t finishing in time so I can adjust the frequency that they sync or select a faster server to rsync from. Occasionally the server I’m rsyncing from will hitting their max connections setting which I like to know about since it means our mirror isn’t getting its updates.
- Be prepared for a lot of traffic – Our server usually peaks around 40-60 Mbps on a normal day but when there’s a CentOS update like from 5.8 to 5.9, we can do 10x that traffic. See the below image of when CentOS 5.9 was released and we peaked at over 400 Mbps, you can see right about midnight, everyone with a daily cron job that does a yum update began updating their servers.
We’re in the process of upgrading our mirror server’s storage space so we can mirror more things so check back in a few weeks to see the new stuff.Posted in: Apache, CentOS, Linux, Nexcess, Software