Ack is Better Than Grep

All posts by Mark

Jan 12

  • Created: Jan 12, 2012 1:33 PM

Ack is Better Than Grep

Ack is Better Than Grep

In my last blog posting I wrote about some basic features of grep. If you already use grep a lot, it was probably a pretty boring post. One problem I used to have is when running grep, I’d get a lot of unwanted files that would match. For example I might try searching for the string ‘foobar’ to find where that appears in some code but there’s a DB dump in the directory too and the .sql file has the word ‘foobar’ in it somewhere. And when it does, it will be on one line that has 300 characters on it which fills my screen with its long line, making things hard to skim through.

Read more

Posted in: Linux
Dec 5

  • Created: Dec 5, 2011 9:45 AM

Features of Grep

Features of grep

In my last blog posting I wrote about features of the less pager. Its one of those very common Linux utilities that everyone uses but no one reads the man page to learn about features. Just like less, everyone uses grep, but there’s a lot of features you’re missing out on if you haven’t read the man page.

grep is a very very old command, its name comes from the command that was run through the ed editor. If you wanted to print out the lines of a file that matched a regular expression (re) you’d run ‘g/re/p’ in ed which stood for globally search for a regular expression and print the matching lines. vi and sed users might recognize the syntax and other basic ed commands since they both adopted a lot from ed.

Read more

Posted in: Linux
Oct 30

  • Created: Oct 30, 2011 7:17 PM

Features of the Less Pager

Features of the Less Pager

less is what is known as a pager in the world of Linux. Pagers are used to view files or output piped in. As their name implies, they’re used when there are multiple pages of text that can’t fit on the screen. Some people use their favorite editor to read log files but I always use a pager because pagers are read-only, there’s no possibility you’ll accidentally make a change to a log file and save it like there is with your favorite editor.

less was written in 1983 by Mark Nudelman, who still maintains it today. The name of it is a play on the name of an older pager called more which is unable to move backwards if you were viewing something piped to it. It can go backwards in a file but not if something is piped in to it, compare more /var/log/dmesg which will allow you to go backwards but not cat /var/log/dmesg | more.

Read more

Posted in: Linux
Sep 26

  • Created: Sep 26, 2011 9:48 AM

Swaks Makes Testing SMTP Servers Easier

Swaks Makes Testing SMTP Servers Easier

I have the pleasure of dealing with mail problems frequently, specifically problems getting messages delivered from our servers to other servers.

When diagnosing a problem, I usually ended up telnetting directly to the server to send it some commands to figure out where it’s failing and why. To do this, I used to run dig mx foobar.com then telnet to the mail server on port 25 and start sending it “EHLO”, “MAIL FROM”, “RCPT” and so on.

Read more

Posted in: Security
Aug 22

  • Created: Aug 22, 2011 12:10 PM

Making the CentOS 6 Boot Splash Screen More Verbose

Making the CentOS 6 Boot Splash Screen More Verbose

CentOS 6 comes with a splash screen that displays a progress bar as it boots. This looks nice and might be cool on a desktop for some eye candy but I’d rather watch what is happening. You can hit any key during the boot process to make the boot splash screen disappear and display what its doing when its booting but that’s annoying and I’d rather it have it spit out all the gory details of what the server is doing automatically without human intervention.

Splash Screen

To make CentOS 6 display the details about what its doing while it boots, first make a backup of the file at /etc/grub.conf in case something goes wrong. Then open /etc/grub.conf in your favorite editor, and look for the line(s) that begin with ‘kernel’. At the end of them you’ll see ‘rhgb’ and ‘quiet’. You’ll want to remove both of those words from grub.conf. After saving your changes, reboot the server and you can see everything its doing when it starts up.

Here’s an example of a grub.conf that has ‘rhgb’ and ‘quiet’ in it:

<br />
title CentOS Linux (2.6.32-71.29.1.el6.x86_64)<br />
	root (hd0,0)<br />
	kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=UUID=c209fbd2-0738-4672-b225-6a5c09f65ad2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet<br />
	initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img<br />

Posted in: CentOS / Tagged: , ,
Jul 15

  • Created: Jul 15, 2011 1:47 PM

Removing old Kernels on CentOS 4

Removing old kernels on CentOS 4

Kernels RPMs are install only packages but CentOS 4 has no limit on the number of install only packages it will keep around, so you can easily end up with years worth of kernels installed on your server. This doesn’t cause any harm but it can be annoying to have so many un-needed packages installed and they can take up space.

You can see all the kernels installed with the command below. The example below has 39 regular kernels and 39 SMP kernels installed. Read more

Posted in: CentOS / Tagged: , , , ,
May 9

  • Created: May 9, 2011 12:05 PM

Using qmail/qmqtool One Liners

qmail/qmqtool One Liners
Qmail is the mail client that comes with the control panel we use, Interworx. Qmail has its strengths and weaknesses, one drawback is there’s no easy built in way to view the messages in the mail queue. One of the ones tools that comes with Interworx ‘s qmail is qmqtool. I’ve found this tool very useful for working on mail queues and debugging problems and have developed a number of one liners to quickly diagnose problems.

<br />
# who are the top senders for the remote queue<br />
/var/qmail/bin/qmqtool -R | grep &quot;From: &quot; | sort  | uniq -c | sort -rn | head -20</p>
<p># who are the top recipients of the remote queue<br />
/var/qmail/bin/qmqtool -R | grep &quot;To: &quot; | sort  | uniq -c | sort -rn | head -20<br />
/var/qmail/bin/qmqtool -R | grep &quot;Envelope Recipient: &quot; | sort  | uniq -c | sort -rn | head -20</p>
<p># what are the top subjects of the remote queue<br />
/var/qmail/bin/qmqtool -R | grep &quot;Subject: &quot; | sort  | uniq -c | sort -rn | head -20</p>
<p># what domains are being sent the most mail from the server<br />
/var/qmail/bin/qmqtool -R | grep &quot;To: &quot; | cut -d @ -f2  | tr -d '&gt;' | sort | uniq -c | sort -rn | head -20<br />
/var/qmail/bin/qmqtool -R | grep &quot;Envelope Recipient: &quot; | cut -d @ -f2  | awk '{print $1}' | tr -d '&gt;' | sort | uniq -c | sort -rn | head -20</p>
<p># who are the top receivers for the local queue<br />
/var/qmail/bin/qmqtool -L | grep &quot;To: &quot; | sort  | uniq -c | sort -rn | head -20<br />
/var/qmail/bin/qmqtool -L | grep &quot;Envelope Recipient: &quot; | sort  | uniq -c | sort -rn | head -20</p>
<p># what domains are have the most mail in the local queue<br />
/var/qmail/bin/qmqtool -L | grep &quot;To: &quot; | cut -d @ -f2  | tr -d '&gt;' | sort | uniq -c | sort -rn | head -20<br />
/var/qmail/bin/qmqtool -L | grep &quot;Envelope Recipient: &quot; | cut -d @ -f2  | awk '{print $1}' | tr -d '&gt;' | sort | uniq -c | sort -rn | head -20<br />

Posted in: Linux / Tagged: , , ,
Apr 2

  • Created: Apr 2, 2011 3:15 PM

Magento Catalog Search XSS on Some Themes

Magento Catalogsearch XSS On Some Themes
We’ve seen a few customers failing PCI scans due to Cross Site Scripting (XSS) vulnerabilities on the catalogsearch page. Initially it seemed like it was a bug in Magento where it wasn’t sanitizing its input but when we tried to see if a new install, using the default theme, was vulnerable we found it wasn’t. This led us to start tracking down the code that actually printed out the XSS.

In the cases we’ve dealt with, the XSS vulnerability is usually reported on the ‘catalogsearch’ page. On the search page it will show info about what your search query was which is where the XSS vulnerability lies. The code that controls this is in breadcrumbs.phtml.

Read more

Feb 25

  • Created: Feb 25, 2011 5:35 PM

Python 2.6 for CentOS 5

Python for CentOS
Python is a great language but CentOS 5 comes with Python 2.4 which is pretty old nowadays. CentOS’s will remain on that version of Python until it reaches EOL since upgrading to a newer versions can introduce changes which are not backwards compatible and cause breakage for applications built for Python 2.4 such as the version of yum that comes with CentOS 5. CentOS provides great stability but it sometimes comes at the cost of missing features as newer versions of software is released.

Googling python 2.6 for CentOS 5 turns up a lot of ways to install it. Some people recommend grabbing the tarball and doing an alt install so it doesn’t overwrite the system python which works but if you want to do everything via RPMs it makes you cringe. Others have compiled Read more

Posted in: Linux / Tagged: , , , ,
Jan 21

  • Created: Jan 21, 2011 4:45 PM

One Liners for Apache Log Files

Apache One-Liners

I frequently need to look at apache log files to diagnose problems. Over time I’ve developed a series of one liners I can copy and paste to quickly analyze a log file to look for a problems, abuse, popular pages, etc.

If someone is reporting a slow site, it can be useful to see if one IP is accesing URLs much more than other IPs since this can be an indication of a poorly written crawler which is using up lots of resources. Other times a slow site might be because someone is getting high traffic so it can be useful to look at the top referrers to see where they’re linked or to look at the most popular URLs and cache that page.

Read more

Posted in: Apache / Tagged: , , ,