Welcome, Linux and Security interested visitors!

My name's Arturo Busleiman. also known as Buanzo by the FOSS and IT Security communities.

I am an old-school GNU/Linux programmer, user, system administrator and enthusiast. I started with it when I was almost 13 years old (late 1994), and I am now 24 (Feb-2006).

As you can probably imagine from that statement, I *love* GNU, Linux, System Design, Deployment and Administration, and of course Security and Programming. I do actually make a living out of my knowledge, and I've decided to open this Blog not only to sell my services, but also to share my knowledge of technology to help out people.

Well, I've been doing exactly that in my Spanish-based web site and Blog, but two months ago I put up an English based page on my site with some stuff I've done in English, and the response to it has been outstanding, that's why I'm also starting this english-only blog. I hope my English is good enough so that you understand every piece of information I publish here.

I expect to provide you with interesting material regarding Anti-Spam solutions, Firewalling techniques, Linux Distributions Downloading and Installing, Linux Programming and all things related to Linux and the Free Software / Open Source movements.

For my first post, here you have my "Detecting and Understanding Rootkits" article, (licensed under the GNU Free Documentation License) which is some years old now, but I think it will make a good start-up.

Thanks for your time!

Yours,
Arturo 'Buanzo' Busleiman


Detecting and Understading rootkits,
an Introduction and just a little-bit-more

by Arturo 'Buanzo' Busleiman
September, 2003.


Well, well, well. You have installed the latest Linux distribution and stopped all unnecessary services. You also set-up a set of Netfilter rules that would make the Pentagon Security epartment envy you. You drool with delight.

But...

There's always one. You should remember this: there is ALWAYS, at least, one flaw. Just one hole through which a cracker could get into your system and do things you would not do to your worst enemy's laptop.

Talking seriously, the Golden Rule of Security is: "Security is just a state of mind". That means that you THINK you are secured. When you come to believe such a thing, you automatically stop worrying. And that is a Bad Thing (tm).

This is, I am sad to say, what a lot of system administrators do. And that is something attackers tend to take advantage of.

The first thing a cracker will do is among one of these: set up sniffers, read eMail and crontab files, then try to discover the "when's and how's" of the system administrator, etc. They, of course, try to hide their tracks. But most of them just go and install a Rootkit.

What is a rootkit?

For me, it is the evolution of the Trojan Horse concept. It is, in these days, a complete package of trojanized system utilities, with some interestingadd-on programs, like specially designed sniffers and, maybe the most dangerous or frightening, kernel modules whose primary objective is to hide certain processes, directories and/or files. Being at the kernel-level can be quite amusing. Imagine: it is the kernel which gives the ability to execute programs and manage filesystem security.

As system utilities and kernels evolve, so do rootkits. Especially the ones that make use of kernel modules. These are called LKM rootkits. Most rootkits used to be packaged as a set of pre-compiled binaries and an installation script that overwrite files.

As time went on, rootkits started to be a bit more complex at the installation stage: they included the source of the trojan utilities and kernel modules. That gave the attacker the ability to analyze the original utilities installed on the system and make the needed modifications to the trojanized ones. This was done to minimize the differences between the original binaries and the trojanized binaries. They also started requesting a "Master Password" that would be inserted into every compiled trojan. The Master Password is used to access the special features of a trojan, like a passwordless root login.

Of course, a C compiler and a complete set of header files are needed. One way to thwart installation of these rootkits is to remove all development packages from a production system.

In any event, if the attacker now has the desired UID 0 then he can download and install the needed packages, or just use a pre-compiled rootkit. In both cases there are disadvantages to the cracker. But those disadvantages are advantages from the system administrator's point of view.


Detecting new rootkits

All of this article is written with the GNU/Linux operating system in mind. This is because it is easier to develop for it, as all of its parts, like the kernel, libraries and utilities are open source free software.

A new rootkit can either be one that has never been seen before, or one that uses new technologies or previously unused methods of attack. Or both. And that is where our rootkit detection problems start.

How can we detect rootkits? There are simple and complex pseudo-solutions. I say "pseudo" because of the number of false positives we may get. I believe the probability of detecting a new rootkit in our system is directly proportional to our knowledge of the subject. Or, in other words, we should know about how rootkits behave.

As I will show later, there are utilities you can use to detect rootkits. These utilities, which are good enough, sometimes give false positives. They can also give false negatives, which in this case is terrible. This is the reason to do behavioural analyses of rootkits. The only way to detect something is to learn about it. It is sometimes a bit of a sixth sense. I know people who can sit down, use the shell a bit, and suddenly they say "I'm pretty sure you've been hacked".

Of course, you also need to know your system. But, well, that is something that only experience gives. In the meantime, I really hope I can help you a bit with this article.

As I stated before, a rootkit is usually a set of:

1) Trojanized system utilities and daemons
2) Additional programs
3) Kernel Modules (now quite common)

Let's look at each individually.

(1) Trojanized System Utilities and Daemons

Some examples and explanations:

ls: usually enables the attacker to hide files and directories. It uses a special configuration file.

du: again, hides certain directories and does not count them against disk usage. It also uses a configuration file.

df: You guessed it! It gives more free space than you actually have, to conceal the fact that a large part of the disk is being used to store sniffer logs.

login: Gives automatic root access, usually after a special trigger is activated. What is special about this root access through login is that loggin can be disabled. It also allows direct login to users other than root.

netstat: Hides connections, based upon system variables such as socket pathname, uid, etc.

chfn/chsh/passwd: Usually has a hidden uid 0 shell.

inetd/xinetd: Usually binds an interactive shell to certain port. Most of them can make use of access control features.

ifconfig: If a sniffer is running, it hides the fact that a network interface is probably in promiscuous mode.

ps/top: Filters out processes from the listing, based on paremeters read from a configuration file. Actually, if the proc pseudo filesystem were unmounted, and a trojanized proc-like filesystem module is loaded and then mounted, one could control a wide range of system utilities that read required information from /proc.

syslogd: It will not write messages which contain configured substrings or regular expressions.

md5sum: It returns a certain md5 hash for certain file. Completely configurable. Scripts that use this binary to verify filesystem integrity can be easily compromised this way.

rshd: (Not used very much anymore) Gives the ability to bind an interactive root shell.

At this point you probably get the idea. As you can see, the pattern that rootkit programmers follow for Trojanized Utilities looks like this:

a) Divide utilities into groups, or think about categories and then find which utilities fit them.

b) Add trojan code based upon the category. This means that a utility that sits in the "Login and/or Remote Access" category will probably have trojan code to enable passwordless root login, for example. The same way, another utility in the "Kernel-Related" category, like lsmod, will not show certain kernel modules in the loaded kernel modules list lsmod provides by reading /proc/modules.

Of course, this process can be applied to any installed program or utility.

As you can probably imagine, utilities that are used locally (i.e, when the user is logged onto the system, either via network or at the console) at the shell to show certain vital or important information (e.g, ps) are potential victims. These kinds of utilities are frequently used by users and, especially, sysadmins.

(2) Additional Programs

These are additional programs used by the attacker when he is logged onto the compromised system. Among these utilities we can, and probably will, find sniffers, zappers, encryption utilities, rootkit post-installation configuration programs, file transmission utilities using a special protocol (like ICMP encapsulation) and the like.

It is easy to figure out that these files need to be kept on the compromised system for them to be worth using. The same applies for files generated by programs like sniffers: sniffer logs do take a lot of disk space if not properly tuned.

Usually, the directory where files will reside is going to be well protected. For example, the rootkit can be configured only to allow certain UIDs to get into certain directories. In other circumstances it will hide the directories. This is achieved by: (1) Using trojanized ls/cd/etc commands or by using a Linux Kernel Module (3). The second approach was some kind of a revolution. Fewer infected files means a lower probability of detection. And since using a kernel module gives a lot of power, a lot of additional features can be conceived.

Usually, rootkit sniffers are developed with the "go to the background and make no noise" approach. This means that they sniff traffic, especially passwords, but they hide the fact that they are running, usually by changing the name that appears on a ps. Trojanized ps/top utilities or /proc filesystems are good approaches to hide processes based on certain configuration parameteres.

Then you have wtmp/utmp/lastlog zappers. As you probably know, these are the files read by the w and last utilities. Early rootkit zappers used to zero the records you told them to (i.e which user to zero-overwrite). Current ones just physically remove the record, instead of logically overwriting it with
zeroes. Some crackers forget that these are not the only places where their tracks can appear.

Of course, there are a whole bunch of additional utilities, like encryption/decryption utilities, but now there are steganography techniques that are starting to be used. Steganography hides files inside other files, by actually replacing the lower bits of an image or MP3 file, for example. One cannot usually see (in the case of an image) or hear (mp3 file) the difference between the real and the 'injected' version of the file. Of course, the original file size grows, sometimes quite noticeably.

File transmission utilities that help the attacker get files out of the system via methods that do not make system noise (i.e log files being written) are not that common, but are still pretty interesting. I remember when I coded a proof of concept utility that would allow me to send files over a ping packet. From a traffic point of view, I was just pinging some remote system. From my point of view, I was transmitting a file. In a very slow, but largely undetectable, way.

(3) Kernel Modules

The main difference between a normal rootkit and an LKM Rootkit is very simple: normal rootkits replace system utilities that enable the attacker to hide files, processes and network connections.

An LKM Rootkit, on the other hand, does something a bit more interesting: it replaces the location of system calls, changing the original memory addresses to something else, and in that different location there is a trojanized version of the system call. So, they do not need to modify utilities (or libraries), they simply replace what these utilities and libraries use! Rootkits of this sort go by the names of Rkit and Adore LKM, just to mention a couple of the most common ones.

Here is a list of the typically modified system calls: sys_clone, sys_close, sys_execve, sys_fork, sys_ioctl, sys_kill, sys_mkdir, sys_read, sys_readdir, sys_write.

Utilities to Detect Rootkits

* Chkrootkit

Rootkit detection utilities are usually a package of many utilities: promiscuous mode detectors for NICs, utilities to detect removed entries from lastlog and wtmp files, some system utility replacements (e.g. strings and find) and utilities to search for sniffer and rootkit log files. In my opinion, chkrootkit is one of the best packages out there: it's small, less than 40kb, and complete. It is very useful and detects many rootkits, like t0rn, lrk, Ark, both versions of Adore, T.R.K, etcetera.

Using it is as simple as "download, untar, make and execute" in most cases, but Chkrootkit provides two command line switches that I find extremely useful:

a) If you can get to the disk you want to analyze and mount it on a trustworthy machine, the -r command line switch allows you to set up the "root" to another directory. For example, if you mount the possibly compromised system on /mnt/testing, you could just use "chkrootkit -r /mnt/testing".

b) If you cannot do this, you should always have trustworthy, statically-linked copies of the following commands: awk, cut, egrep, find, head, id, ls, netstat, ps, strings, sed, and uname. Chkrootkit makes use of these system utilities to search and analyze binary and log files. As these system utilities may already be compromised, it would not be a good idea to use them. Provided you have these safe utilities in /mnt/safebin, you can use "chkrootkit -p /mnt/safebin". You can specify multiple directories, just by separating them with colons. Of course, it would be a good idea to keep these safe binaries on a read-only medium, like a CD-ROM or a protected floppy disk.

Of course, you can use both the -r and -p switches in combination to provide the maximum poddiblr level of safety.

Chkrootkit has an expert flag, -x, which outputs strings taken from the binary files it analyzes. It generates a lot of output, so it is a good idea to redirect all of it to a file so it can be throughly analyzed.

* LKM-Rootkit detection

The only way an LKM rootkit can be detected is by analyzing kernel memory directly. One of way to do this is to compare system call addresses (you will recall that LKM rootkits change them). This task can be easily performed by using tools such as kstat, which read kernel memory through /dev/kmem. kstat provides information on running processes via its '-P' switch, which includes hidden processes. Compare its output with what "ps aef" tells you. Additionaly, you can query a specific process id with the '-p ' parameter. To analyze system call addresses you should specify the '-s' switch. After an initial system installation and full configuration, record "kstat -s" output. Memory addresses there will provide correct values you can compare from time to time. Lines with a WARNING show the possibility that your system has been compromised. kstat can also act as a replacement for lsmod with the '-M' switch. You will be able to read the trojan kernel module on the list.

* Tripwire and similar tools

If well configured, Tripwire can be an excellent tool. It uses GnuPG-like encryption and signing of database files. These database files contain hashes and other data like permissions and ownership for system files and directories. A system administrator can execute periodic tests against this database and, if differences appear between the stored and the current status of the filesystem, Tripwire will send a very detailed report through eMail.

Of course, you could use a script to generate an initial md5 hash database to be stored on a CD-ROM, along with the required md5sum utility (statically linked). You could also add signature-checking capabilities through GnuPG. I prefer this method over Tripwire, because I prefer doing things simply. Tripwire requires a lot of effort, whereas you can use this method by using built-in utilities.

* Port Scanning

Most rootkits open up an administrative port, or bind a remote shell. Because of this, it would be a good idea to port scan the full range of TCP and UDP ports, from 1 to 65535, and mix that output with "kstat -P" and "lsof -i", so you can recognize which processes open which files/sockets. One of the best tools out there, as we can see on Matrix Reloaded, is Fyodor's most excellent Nmap. An external firewall (I mean a real firewall, not a Netfilter on the same host), could be used to block traffic to these ports, but this method usually causes big arguments. As Einstein would say: All is relative.

Dealing With Rootkits Once You Find Them

It all depends, but this is usually the simplest step. This article is intended for System Administrators, and not Honeynets or security developers. The usual thing to do would be to re-install the system, replacing the trojan files. Do not trust backups at all, unless you are absolutely and completely sure that they are not infected. If possible, dump all of your databases, all of your data, and do a fresh install, instead of an overwriting one.

But you should always try to discover through which hole the attacker got in. This way they will not be able to do it again after the system reinstallation. Remember the Golden Rule. Remember to update packages, either by compiling them yourself, which is what I preffer, or via your vendor's Online Update service.

Final Comments

The only way to detect something is to understand what it does, how it looks, and how it does what it does. For rootkits and LKM rootkits, this means that we should know how they work (i.e trojanizing binaries or system calls), how they look (a badly installed rootkit can make a system unstable or unusable; the t0rn rootkit has a tendency to break console logins if not correctly set up), and how they do it (replacing files or system calls addresses). Setting up an Intrusion Detection System, like Snort, would be a good idea.

I hope you have found this article useful. Please access the following URLs to find commented software and additional documentation.

Greetings from Buenos Aires, Argentina!

URL list:

* Netfilter: http://www.netfilter.org - Linux 2.4 firewalling, NAT and packet
mangling subsystem and iptables user-space tools.
* Sniffers: Ettercap. http://ettercap.sf.net - An excellent modularized
sniffer, both for interactive and non-interactive usage.
* GNU/Linux: http://www.gnu.org - FSF's GNU Project.
http://www.kernel.org - The Linux Kernel
* Cryptography and Steganography: http://www.topology.org/soft/crypto.html -
any URLs you will enjoy.
* Chkrootkit: http://www.chkrootkit.org - Chkrootkit and additional
documentation.
* Kstat: http://s0ftpj.org/en/site.html - 'soft' with zero, not o.
* Tripwire: http://www.tripwire.org
* Honeynets: http://www.honeynet.org - The Honeynet Project. Highly Recommended. Quote from site: "To learn the tools, tactics, and motives of the blackhat community, and share those lessons learned."
* Nmap: http://www.insecure.org/nmap/
* Snort: http://www.snort.org - IDS System.