Sys::Links

I've Moved the official blog to Sys-Log.BenCane.com but I will still be using tumblr as a kind of link grab for my articles as well as other cool articles on the interwebs.

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

For todays article I am going to explain how to create a basic firewall allow and deny filter list using the iptables package. We will be focused on creating a filtering ruleset for a basic everyday Linux web server running Web, FTP, SSH, MySQL, and DNS services.

Before we begin lets get an understanding of iptables and firewall filtering in general.

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

[sys-log-blog-blog]

When Zombies Invade Linux: What are Zombie Processes and What to do about them

Zombies don't just appear in scary movies anymore, sometimes they also appear on your Linux systems; but don't fret they are mostly harmless.

What is a Zombie Process?

Before we get started I wanted to first cover what exactly a Zombie process is. Linux and Unix both have the ability for a process to create a sub process otherwise known as a "Child Process". Once a process creates a new sub process the first process then becomes a "Parent Process" as it has spawned a child process during its execution.

A Zombie or defunct process is a process that has finished its execution and is waiting for its Parent Process to read its exit status. Because the child process has finished, it is technically a "dead" process however since it is waiting for its parent there is still an entry in the process table. The zombie's parent process does not necessarily need to be running for a zombie to appear, however it is most common to see a zombie process whose parent has died unexpectedly.

How to spot a Zombie Process

Zombie processes can be found easily with the ps command. Within the ps output there is a STAT column which will show the processes current status, a zombie process will have Z as the status. In addition to the STAT column zombies commonly have the words <defunct> in the CMD column as well. Example:

$ ps -elf | grep Z F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 1 Z madflojo 28827 28821 0 80 0 - 0 exit 12:28 pts/4 00:00:00 [zombies.aahhh] 1 Z madflojo 28828 28821 0 80 0 - 0 exit 12:28 pts/4 00:00:00 [zombies.aahhh] 1 Z madflojo 28829 28821 0 80 0 - 0 exit 12:28 pts/4 00:00:00 [zombies.aahhh] 1 Z madflojo 28830 28821 0 80 0 - 0 exit 12:28 pts/4 00:00:00 [zombies.aahhh] 1 Z madflojo 28831 28821 0 80 0 - 0 exit 12:28 pts/4 00:00:00 [zombies.aahhh]

What is the difference between a Zombie and Orphaned Process?

Orphaned processes are very similar to Zombie processes; however there is one major difference. An Orphaned process is a child process that is still an active process whose parent has died. Unlike zombies the orphaned process will be reclaimed or adopted by the init process.

How to spot an Orphaned Process

Orphaned processes can be found easily with the ps command as well. Within the ps output there is a PPID column which will show the processes parent process id; a orphaned process will have the PPID of 1 which is the init process. You may be thinking to yourself, how do I differentiate an Orphaned process from a Daemon process? Well in short, there is no difference. For all intents and purposes a daemon process is a orphaned process, however the exiting of the parent process is on purpose rather than by error. Example:

$ ps -elf | grep sshd 4 S root 718 1 0 80 0 - 12487 poll_s Jun07 ? 00:00:00 /usr/sbin/sshd -D

What to do about Zombie Processes?

Before performing any activity to clean up zombie processes it is best to identify the root cause of the issue. Zombie processes do not indicate a normal state for your system, they may be benign for now however like real zombies they become more troublesome when they are in large numbers. They also indicate either a system issue or an application issue depending on the source of the processes. The steps necessary to clean up zombie processes is complicated and very situational, below are a couple of high level answers that can guide you to a solution.

If the parent process is still active

If the parent process of the zombie or zombies is still active (not process id 1) than this is an indication that the parent process is stalled on a certain task and has not yet read the exit status of the child processes. At this point the resolution is extremely situational, you can use the strace command to attach to the parent process and troubleshoot from there. You may also be able to make the parent process exit cleanly taking its zombie children with it by issuing the kill command. If you do run the kill command I suggest that you run a kill with the default signal -15 (SIGTERM)rather than using a -9 (SIGKILL); as SIGTERM will tell the parent process to exit cleanly which is more likely to read the exit status of the zombie children.

If the parent process is no longer active

If  the parent process is no longer active than the clean up activity becomes a choice; at this point you can leave the zombie processes on your system, or you can simply reboot. A Zombie process whose parent is no longer active is not going to be cleaned up without rebooting the system. If the zombie processes are only in small numbers and not reoccurring or multiplying than it may be best to leave these processes be until the next reboot. If however they are multiplying or in a large number than this is an indication that there is a significant issue with your system.

[sys-log-blog-blog]

ACL: Using Access Control Lists on Linux

Access Control Lists aka ACL's are one of those obscure Linux tools that isn't used every day; and if you find yourself using ACL's every day than you probably have a very complicated Linux environment.

A few years ago I had an engineer tell me "Any thing you want to solve with ACL's can be solved with standard unix permissions" and while he may have just been justifying why he didn't know ACL's very well.

He did have a point; kind of. In general ACL's are a bit of an advanced topic for Unix/Linux systems administrators and unless you have a shop full of experienced sysadmins and users than it is usually best to reserve ACL's for when there is no better option. With that said, I have found several occasions where ACL's were needed and it is always a good idea to learn advanced topics even if they aren't used in everyday situations.

This article can be used as a tutorial for ACL's on Linux; I will be covering the basics of ACL's and give some examples of usage.

Prerequisites

Filesystem options

Before using ACL's we must first verify that our filesystem has the acl option enabled. A common way to enable acl support on a filesystem is to add the acl option to a filesystems mount options in /etc/fstab. We can check if that has been done on this system by using the mount command.

root@testvm:~# mount | grep root /dev/mapper/workstation-root on / type ext4 (rw,errors=remount-ro)

In this case the acl option has not been added but that doesn't mean our filesystem doesn't have acl's enabled. On most distributions the default filesystems have the acl option as part of the default mount options. You can check if your filesystems have acl as part of the defaults by using the tune2fs command.

root@testvm:~# tune2fs -l /dev/mapper/workstation-root tune2fs 1.42 (29-Nov-2011) Filesystem volume name: <none> Last mounted on: / Filesystem UUID: f558402c-a418-4892-87e2-071c1a85898c Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl

As you can see on my test system the default mount options contain acl, in this case my filesystem will support acl's even I don't specify it during the mount process. If your filesystem does not have acl as a default mount option, than you can add it during the mount process easily by editing the fstab.

root@testvm:~# vi /etc/fstab

Simply add the term acl to the mount options as shown below. Before:

/dev/mapper/workstation-root / ext4 errors=remount-ro 0 1

After:

/dev/mapper/workstation-root / ext4 acl,errors=remount-ro 0 1

Once your fstab file is edited you can remount your filesystem with the mount command.

root@testvm:~# mount -o remount / root@testvm:~# mount | grep root /dev/mapper/workstation-root on / type ext4 (rw,acl,errors=remount-ro)

ACL Utilities

Now that your filesystem supports acl's we must make sure that you have the acl utilities installed. My test machine is an Ubuntu server install so I will use the dpkg command, if you are running a different Linux distro than you may need to use another package manager, consult your distro's documentation.

root@testvm:~# dpkg --list | grep acl ii libacl1 2.2.51-5ubuntu1 Access control list shared library

As you can see on my test machine I have the libraries necessary for acl support but not the command line utilities to interact with acl's. Lets change that by installing the tools with apt-get; again this is part is dependent on the distribution you are using.

root@testvm:~# apt-get install acl

Now we check again, and it is installed.

root@testvm:~# dpkg --list | grep acl ii acl 2.2.51-5ubuntu1 Access control list utilities ii libacl1 2.2.51-5ubuntu1 Access control list shared library

Setting and listing acl's with getfacl and setfacl

Now that we have the utilities installed we can start using the setfacl and getfacl commands.

Setting ACL's with setfacl

The setfacl command is used to set acl lists on files and directories, setfacl is the acl version of the chmod command. As an example I am going to give the testusers group permission to read from the appdir which is owned by root:appgroup.

root@testvm:/var/tmp# ls -la | grep appdir drwxrwxr-x 2 root appgroup 4096 May 27 10:45 appdir

To add these permissions we will use the setfacl command

root@testvm:/var/tmp# setfacl -m g:testusers:r appdir/

Let's break the command down a little bit.

-m

The -m option tells setfacl to modify the acl list for the specified directory.

g:testusers:r

This is actually the access control list that is being set. The first column is specifying g for group, the second column is the group name that I want the permissions to be set for and the last column is the permissions I want that group to have. In this case the read permission.

appdir/

This is the directory that I am setting the permissions on.

Checking the ACL list with getfacl

Once you've set the acl with setfacl it is common sense to check if it took effect, in order to do so you will need to use the getfacl command.

root@testvm:/var/tmp# getfacl appdir/ # file: appdir/ # owner: root # group: appgroup user::rwx group::rwx group:testusers:r-- mask::rwx other::r-x

The output of getfacl is pretty self explanatory, you can see the rule that we added below the standard group entry.

Identifying files/directories that have ACL's

While the standard unix permissions are displayed with the ls -l command; the defined ACL's are a little more verbose and are not a part of the long listing. The command ls will tell you if a file or directory does have acl's, it's just not that obvious.

root@testvm:/var/tmp# ls -la | grep appdir drwxrwxr-x+ 2 root appgroup 4096 May 27 10:45 appdir

As you can see there is now a + at the end of the directories permissions. This + is the indicator that this file or directory has acl's, from here you can use the getfacl command to see what they are.

Examples of Usage

Now that we have a filesystem that supports ACL's and we know how to set and review the ACL's lets run through a few examples of ACL usage.

Use Cases

Removing all acl entries from a file or directory

Set test users to have read access to all files in a directory

Set the same acl changes recursively

Set the same acls on all newly created files automatically

Set testuser1 to have read, write and execute access to the appuser1 directory

Set all users to have read, write and execute access to the shared directory

Remove the acl for testuser1 on appuser1 directory

Removing all acl entries from a file or directory

Before we start messing with acl's in my directory I want to clear out all of the acl's it previously had. Doing this one by one can be a bit of a pain, its a good thing the setfacl command gives you the ability to remove all acl's on a specified file or directory. This can be accomplished using the -b option of setfacl.

root@testvm:/var/tmp# getfacl appdir/ # file: appdir/ # owner: root # group: appgroup user::rwx group::rwx group:testusers:r-- mask::rwx other::r-x root@testvm:/var/tmp# setfacl -b appdir/ root@testvm:/var/tmp# getfacl appdir/ # file: appdir/ # owner: root # group: appgroup user::rwx group::rwx other::r-x

Keep in mind when using the -b option that this will remove all of the acl rules on the specified directory.

Set testusers to have read access to all files in the appdir directory

This is a pretty basic acl, we want the testusers group to have read access to all files in the appdir directory.

root@testvm:/var/tmp# setfacl -m g:testusers:r appdir/ root@testvm:/var/tmp# getfacl appdir # file: appdir # owner: root # group: appgroup user::rwx group::rwx group:testusers:r-- mask::rwx other::r-x

Set the changes recursively

The appdir has 2 sub directories appdir/appuser1 and appdir/appuser2 in this case we want the acl rules set above to apply to these directories as well. This can easily be accomplished by adding the -R (recursive) option in setfacl. This works exactly like the recursive option for chmod.

root@testvm:/var/tmp# setfacl -Rm g:testusers:r appdir/ root@testvm:/var/tmp# getfacl appdir/* # file: appdir/appuser1 # owner: appuser1 # group: appgroup user::rwx group::r-x group:testusers:r-- mask::r-x other::r-x # file: appdir/appuser2 # owner: appuser2 # group: appgroup user::rwx group::r-x group:testusers:r-- mask::r-x other::r-x

Set the same acl's on all newly created files automatically

The -d (default) option in setfacl is extremely useful; this option will allow us to set an acl rule to be the default rule. When this is set on a directory this makes all new files or directories created within that directory inherit the same acl rules.

root@testvm:/var/tmp# setfacl -dm g:testusers:r appdir/ root@testvm:/var/tmp# touch appdir/file1 root@testvm:/var/tmp# mkdir appdir/dir1 root@testvm:/var/tmp# getfacl appdir/* # file: appdir/appuser1 # owner: appuser1 # group: appgroup user::rwx group::r-x group:testusers:r-- mask::r-x other::r-x # file: appdir/appuser2 # owner: appuser2 # group: appgroup user::rwx group::r-x group:testusers:r-- mask::r-x other::r-x # file: appdir/dir1 # owner: root # group: root user::rwx group::rwx group:testusers:r-- mask::rwx other::r-x default:user::rwx default:group::rwx default:group:testusers:r-- default:mask::rwx default:other::r-x # file: appdir/file1 # owner: root # group: root user::rw- group::rwx #effective:rw- group:testusers:r-- mask::rw- other::r--

As you can see dir1 also have the default acl rules as appdir, yet appuser1 does not.  This is because we did not set the default recursively; this can be done by using the -R option.

root@testvm:/var/tmp# setfacl -Rdm g:testusers:r appdir/ root@testvm:/var/tmp# getfacl appdir/appuser1 # file: appdir/appuser1 # owner: appuser1 # group: appgroup user::rwx group::r-x group:testusers:r-- mask::r-x other::r-x default:user::rwx default:group::r-x default:group:testusers:r-- default:mask::r-x default:other::r-x

Set testuser1 to have read, write and execute access to the appuser1 directory

While the user testuser1 is in the testusers group and has read access to the appuser1 directory he does not have write access. In this case we want to give him write access without giving the rest of the testusers write access. This can be done using acl's by specifying a specific user rather than a group.

root@testvm:/var/tmp# setfacl -m u:testuser1:rwx appdir/ root@testvm:/var/tmp# getfacl appdir/ # file: appdir/ # owner: root # group: appgroup user::rwx user:testuser1:rwx group::rwx group:testusers:r-- mask::rwx other::r-x default:user::rwx default:group::rwx default:group:testusers:r-- default:mask::rwx default:other::r-x

The user testuser1 can now create files in the appdir1 directory.

root@testvm:/var/tmp# sudo -u testuser1 touch appdir/file2 root@testvm:/var/tmp# ls -la appdir/file2 -rw-rw-r--+ 1 testuser1 testusers 0 May 27 12:17 appdir/file2

Set all users to have read, write and execute to the shared directory

We have now given users and groups permissions on directories and files, but what happens when we want all users to have access to a directory? Adding every users name or group could get tedious, in this case we can set the "other" or "world" permissions so that all users on a system can access this directory.

root@testvm:/var/tmp# setfacl -m o::rwx shared root@testvm:/var/tmp# getfacl shared/ # file: shared/ # owner: root # group: root user::rwx group::r-x other::rwx

You might be asking yourself right now "wait a minute, did setfacl just set the permissions to 757?"; why yes it did! ACL's are just an extension of the standard unix permissions, in this case because we are not specifying a user or group; setfacl will simply just change the mode of the file. Tricky, yes I know.

Remove the acl for testuser1 on appuser1 directory

Unlike the -b option that removes all acl's on a directory or file the -x option will only remove the specified rule. This is useful for when you maybe fat fingered an acl rule and don't want to completely remove all of the acl rules attached to that file/directory.

root@testvm:/var/tmp# setfacl -x u:testuser1 appdir/ root@testvm:/var/tmp# getfacl appdir/ # file: appdir/ # owner: root # group: appgroup user::rwx group::rwx group:testusers:r-- mask::rwx other::r-x default:user::rwx default:group::rwx default:group:testusers:r-- default:mask::rwx default:other::r-x

As you can see setfacl did not remove the group permissions or the default permissions only the acl specified.

Closing Thoughts

By now if you have followed this primer you should have a general idea of how ACL's work with Linux and the commands you need to set and review them. I do want to give a word of advice however; If you find yourself wanting to use ACL's to solve a problem, don't. That is unless you work in a shop where all of the sysadmins and users are experienced Linux users and understand how ACL's work and how to troubleshoot permissions problems with ACL's. If thats the case then go for it and have fun. For the most part, most sysadmin's don't use ACL's everyday like they would normal Unix permissions. With that said if you are the only person who will admin the system and you want to learn and use ACL's, go for it! You can only blame yourself if it all goes wrong.

[sys-log-blog-blog]

Stat: Detailed information about a file

Stat is a command that I never knew about until somewhat recently but afterwards have had more and more reasons to use it. When run against a file stat will show detailed information about the file, this information can be extremely useful and I want to highlight some of the information I've found useful from stat.

$ stat rsync.out File: `rsync.out' Size: 696506 Blocks: 1368 IO Block: 4096 regular file Device: fc00h/64512d Inode: 13862 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/madflojo) Gid: ( 1000/madflojo) Access: 2012-05-21 19:28:00.953777255 -0700 Modify: 2012-05-24 08:30:19.949780228 -0700 Change: 2012-05-24 08:30:19.949780228 -0700

Inode

The stat command shown above displays an inode number of 13862, this is the inode number of my file. It's not often that you need an inode number but to give an example usage if you wanted to compare two files to see if they are a hardlink or not you can check if the inode number is different for each file.

Links 

The links count is a count of how many hardlinks exist for that file. The way hardlinks work is basically by creating another access point for the same inode/file. When you have multiple links the number raises. As a note the file shown above does not have any hard links created for it hence why the links count is only 1, the original file is the only link.

Access Time

Access time has been extremely helpful for me in many situations, it answers questions like is it possible to modify the file without the access time changing. As seen in the example, yes. ls is able to show you the access time as well but by default with a -l it will only show you the change time.

I'm sure there are many other cases where stat comes in handy but for me these 3 seem to be the most common. If you know of any more common use cases post a comment and share it with us.

[sys-log-blog-blog]

Unix Shell: The Art of I/O Redirection

One of the primary tricks in my sysadmin bag-o-tricks is Input/Output Redirection; I have found that while many people use I/O Redirection throughout their day not everyone fully understands why and how it works.

The Input and Output

In the Unix environment there is always 3 streams open stdin, stdout, & stderr; these special streams are used for interacting with the user input and program ouput within the Unix/Linux shell environment. In addition to files in /dev/ these streams also have a reserved file descriptor attached to them; meaning you can interact with them from the command line.

Before we start showing you how to redirect these streams lets first explain what they are.

stdin - Standard Input

This stream is used to get input for commands from the user keyboard. stdin has the file descriptor of 0 and a file of /dev/stdin.

stdout - Standard Output

This stream is used for non-error output from programs. stdout has the file descriptor of 1 and a file of /dev/stdout.

stderr - Standard Error

This stream is used for error output from programs. stderr has the file decriptor of 2 and a file of /dev/stderr.

Examples of Input and Output

To explain the interaction better I am going to use the following text formats, stdin will be bolded, stdout will be italic, and stderr will be underline.

When cat is run without giving it a filename the cat program opens and expects user input.

madflojo@netbook:~$ cat

Whatever text I type with my keyboard is considered Standard Input; the cat command will send my input out as Standard Output.

madflojo@netbook:~$ cat Say hello netbook Say hello netbook

When a program in the Unix environment prints errors or diagnostic information it uses the Standard Error stream rather than Standard Output; This helps prevent mixing a commands errors with the same commands output.

madflojo@netbook:~$ cat ./stuff cat: ./stuff: No such file or directory

Redirection

How is all of this useful? Well the various Input and Output methods become extremely useful when using the redirect features of the Unix/Linux shell programs. For example, what happens when you want to take the output (stdout) of one program and write it to a file? Well you can use that by using special Redirection Operators designated within your shell.

Redirect to a file

program > file

The > (left caret) symbol is used to redirect to a file; but be warned that a single left caret will truncate the file, then enter the redirected data into the file; this effectively replaces your data. If the designated file does not exist the file will be created. The > symbol will redirect stdout by default but you can specify whether to redirect stdout or stderr by putting the appropriate file descriptor in front of the redirect symbol.

1> - Redirects stdout (default) 2> - Redirects stderr

madflojo@netbook:~$ echo test > /tmp/filename madflojo@netbook:~$ cat /tmp/filename test

Redirect to a file but append the data

program >> file

When two > (left caret) symbols are used the output that is being redirected is appended to the file rather than overwriting the file entirely. Again the file descriptor can be used to specify whether to write stdout or stderr to the file specified.

madflojo@netbook:~$ cat ./stuff 2>> /tmp/filename madflojo@netbook:~$ cat /tmp/filename test cat: ./stuff: No such file or directory

 Redirect input from a file

program < file

Much like the > (left caret) the < (right caret) can be used to redirect stdin from a file to a program.

madflojo@netbook:~$ cat < /tmp/filename test cat: ./stuff: No such file or directory

Pipe output from one program to another

program | program

The | (pipe) is used to redirect stdout from one command to stdin for another. The | (pipe) is not considered a Redirect Operator but rather a Control Operator; while | (pipe) is used to redirect output not all Control Operators have this function.

madflojo@netbook:~$ ls /tmp/ | grep filename filename

 Redirect stderr to stdout

program > filename 2>&1

When using redirects you can use the &N (where N = file descriptor) to redirect 2 methods of output to the same path. In my example above I am redirecting the stderr to the file; This can also be used with the | (pipe) control operator.

madflojo@netbook:~$ ls /tm/ /tm2/ 2>&1 | grep 2 ls: cannot access /tm2/: No such file or directory

Write to a file and stdout

program | tee filename

While tee is neither a control or redirection operator I feel it deserves being mentioned. Tee is a command that will take the stdin given to it and write that to both the file specified and to stdout. I find this extremely useful when running a command that you both want to log the output and see the output on your screen.

madflojo@netbook:~$ ls | tee -a filelist.txt Desktop Documents Downloads madflojo@netbook:~$ ls -la filelist.txt -rw-rw-r-- 1 madflojo madflojo 113 2012-04-15 22:01 filelist.txt

If the -a flag is added to tee the command will append the output to the file specified rather than overwriting.

[sys-log-blog-blog]

Sysadmin Tools: Managing Linux from a Windows Desktop

While it is getting more common for companies to allow their IT staff to choose their own OS not every company allows this. In fact most companies require their IT staff to use Windows, as Windows has historically been the dominate OS for the business world. While I personally believe it is easier to administer Linux/Unix servers using a Linux desktop this doesn't mean that I get to be the exception to the rule; sometimes I have had to use a Windows desktop. Below is an outline of some various tools that I've used to help manage Linux/Unix servers in a Windows world.

PuTTY Suite

PuTTY | Download Here

PuTTY is the most popular SSH client for Windows. Everyone and anyone who has SSHed into a server from a Windows desktop has used PuTTY, and for good reason. It just works.

I find PuTTY is easy to understand for even the most novice user, feature rich (2 words SSH Tunnels), and open source (Windows & Unix).  I think PuTTY is a great tool and I'm not alone as it has made the top 50 list on Life Hackers: 50 Free Apps We're Most Thankful For.

Pagent | Download Here

While PuTTY is great there are some other tools that help it along as well. Pagent is a SSH key authentication agent, Pagent will allow you to import your RSA or DSA keys and works in conjunction with PuTTY to provide you with key based authentication.

All you have to do is import your private key into Pagent and push your public key to the server/servers you want to login to.

PuTTYgen | Download Here

PuTTYgen works great with Pagent, it allows you to generate an RSA or DSA public/private key pair. You can use this with Pagent or use it just to generate RSA/DSA keys for your servers. I do want to point out that you are not required to use PuTTYgen for Pagent, Pagent will allow you to import private keys that were generated other places including Linux/Unix servers.

Other PuTTY tools

While the PuTTY tools above are distributed by the original developer of PuTTY there are some other tools out there developed by others that help utilize PuTTY even more.

PuTTY Session Manager | Download Here

PuTTY Session Manager is probably one of my favorites; PSM uses the sessions that are saved using PuTTY and allows you to organize them and display them in folder trees within PSM.  This is a great tool if your organization is just getting started and you do not have a database of the servers you manage. The Import/Export features let you use PuTTY Session Manager as your "list of servers" when you only have a few in your inventory.

PSM can launch PuTTY, Filezilla or WinSCP client connections to the server you choose. It also works well with Pagent to use key based authentication.

ClusterSHiSH | Download Here

Ever have the need to run the same command on multiple hosts but don't have a server to kick off an SSH for loop from? What about hundreds of hosts? ClusterSHiSH can help. With only a few key strokes ClusterSHiSH can attach to all or a select few PuTTY windows and run whatever command you want. In the hands of a seasoned Systems Administrator this can be a great tool, in the hands of a sloppy typer this can be a nuclear bomb to your server farm. Discretion is advised.

PuTTY Command Sender can perform the same tasks as ClusterSHiSH however; I personally prefer ClusterSHiSH as you can use a checkbox to select what PuTTY windows you want to run the command in and PuTTY Command Sender uses a REGEX like rule to selectively choose. I feel a checkbox is a lot faster than a REGEX.

PuTTY Command Sender does have a win with the ability to hide password input, whereas ClusterSHiSH will only display its input in plain-text.

PuTTYcyg | Download Here

PuTTYcyg is a patched version of PuTTY that gives you an extra option to login to a local Cygwin terminal, if you are using Cygwin this is a must have.

Linux & Linux-like Environments

Sometimes you just need a local command line environment, here are a couple of tools that can help you get there.

Cygwin | Download Here

I've used Cygwin off and on for the past 4 - 5 years, sometimes I love how close to Linux it is and sometimes I hate that its not exactly the same. If you want a Linux like command line environment on your Windows OS than Cygwin is great and while it is close it is still not exactly Linux.

Oracle VirtualBox | Download Here

When you need the real thing at your finger tips VirtualBox can save the day. VirtualBox is an open source virtualization platform that you can install in your Windows OS and allows you to create and run Linux/Unix/Other virtual machines. When you install VirtualBox Guest Utils you can make your VM full screen and allow copy-paste between your Windows OS and your Virtual Machine.

For the past few years I've been using VirtualBox with an Ubuntu or Fedora virtual machine as I love having the power of Linux at my finger tips. The only downfall of this solution is that your desktop machine needs to have enough memory/cpu power to run the virtual machine inside (not hard to find with today's hardware).

File Transfer Applications

FileZilla | Download Here

FileZilla is a great application, it can transfer files using FTP, FTPS, or SFTP. Whenever I want to get a file off a server and I can't use scp from my Cygwin or Linux VM command line, I use FileZilla.

WinSCP | Download Here

While I personally like FileZilla, I believe that WinSCP has its place. Many companies allow the installation of WinSCP but not FileZilla and sometimes the other way around. In this post I'm not trying to say which one is better, I've used them both and I personally default to FileZilla, but WinSCP is good too.

These are just a few tools I've found extremely helpful, but I am always on the look out for more. If you know of any that I did not mention feel free to post it in the comments section.  

[sys-log-blog-blog]

Advanced Linux Troubleshooting: strace

Today I want to cover one of the best troubleshooting tools in any sysadmins arsenal; strace.

Strace is a command that will trace the system calls and signals from a specified command. What does that mean in layman's terms? Strace will output all of the inner workings of a process you run it against.

If a process opens a file or binds a port, strace will print that action; it is a great utility for troubleshooting when a process is not behaving as expected and you can't find any reason in the commands output or log files.

Below is a real scenario that happened to me a few weeks ago.

The Issue

When trying to start vsftpd with the service command (or systemctl on newer versions) the daemon would fail to start.

[root@laptop ~]# systemctl start vsftpd.service Job failed. See system logs and 'systemctl status' for details.

[root@laptop ~]# systemctl status vsftpd.service vsftpd.service - Vsftpd ftp daemon

Loaded: loaded (/lib/systemd/system/vsftpd.service; disabled) Active: failed since Sat, 24 Mar 2012 11:55:25 -0700; 2min 28s ago Process: 19356 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (code=exited, status=1/FAILURE) CGroup: name=systemd:/system/vsftpd.service

As you can see the output of these commands are not very descriptive as to why vsftpd will not start. So I started looking at the log files; vsftpd does not have many log files configured by default but I checked both the xferlog and the messages just in case I could find a clue as to why it did not start.

After the log files didn't give me anything useful, I started looking at the /etc/vsftpd/vsftpd.conf file because I knew some of my colleagues were tinkering with it on this server recently. After a little review the configuration file looked normal and did not have anything odd in it.

Just in case though I replaced the configuration file with the default file and tried to restart vsftpd; again the service did not start. This told me that my issue was not with the configuration of vsftpd. I decided to try starting vsftpd manually via the command line hoping that will output a useful error (sometimes this works).

[root@laptop ~]# /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf [root@laptop ~]# ps -elf | grep vsftpd 0 S root 19435 19077 0 80 0 - 1147 pipe_w 12:05 pts/1 00:00:00 grep --color=auto vsftpd

As you can see from the output I received even less information from trying to start vsftpd manually.

Using strace for troubleshooting

At this point I called in my trusty friend strace. The syntax of strace is pretty straight forward you simply put strace <command line> however I am going to use the -f flag which allows strace to also display the system calls and singals of child processes started by the command you provided.

[root@laptop ~]# strace -f /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

<truncated output (it is seriously a lot of output)>

[pid 19499] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 [pid 19499] setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 [pid 19499] rt_sigaction(SIGCHLD, {0xf5ac00, ~[RTMIN RT_1], 0}, NULL, 8) = 0 [pid 19499] rt_sigaction(SIGALRM, {0xf5abd0, ~[RTMIN RT_1], 0}, NULL, 8) = 0 [pid 19499] rt_sigaction(SIGHUP, {0xf5ac00, ~[RTMIN RT_1], 0}, NULL, 8) = 0 [pid 19499] rt_sigaction(SIGALRM, {0xf5abd0, ~[RTMIN RT_1], 0}, NULL, 8) = 0 [pid 19499] bind(3, {sa_family=AF_INET, sin_port=htons(21), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use) [pid 19499] fcntl64(0, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) [pid 19499] fcntl64(0, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0 [pid 19499] write(0, "500 OOPS: ", 10) = 10 [pid 19499] write(0, "could not bind listening IPv4 so"..., 36) = 36 [pid 19499] write(0, "\r\n", 2) = 2 [pid 19499] exit_group(1) = ? Process 19499 detached

The output of strace'ing vsftpd is quite a bit and to be honest is pretty confusing for those who are not familiar with the different system calls. But even if you are not familiar with system calls, strace can help you.

When reviewing the output of strace you simply look for words that suggest an error.

As a caveat when you see the process not able to open a file that does not necessarily mean this is the cause of your issue. open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

I highlighted the key messages from my strace above, specifically the message that says "could not bind listening IPv4 so" and "Address already in use".

These messages told me that the vsftpd daemon could not start listening on the local IP with port 21. After checking netstat I found there is another process already listening on port 21.

[root@laptop ~]# netstat -nap | grep :21 tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 19212/vs-something

Since there was already a process listening on port 21, vsftpd could not bind the port and hence it could not start. Though it could have been nicer about telling me why it didn't start!

In my real situation inetd has ftp enabled but I was starting vsftpd with the service command.

After identifying the offending process I killed it (as it shouldn't have been running) and configured the system so that the process would not start again and vsftpd would start normally.