Title: Responding to a security incident on a Unix workstation. $Revision: 1.2 $ $Date: 2002/04/17 15:44:37 $ $Author: dittrich $ Q: I've found that my system has been compromised, used to attack another site, or has a network sniffer installed. What do I do now? A: Your system is out of your control. This is a call to action, but *not* a cause for panic. Unless you were told your system is actively attacking another site you do not need to act immediately. The attackers may have been in your machine for days, weeks, possibly even months, so another few hours won't be that bad. (If your system is actively attacking another site, you may be asked to disconnect it from the network right away, or C&C may be forced to block connectivity to the Internet to stop the attack.) Consider what you must do ========================= What you need to do right now is understand the steps necessary to regain control of your system, prepare for and take those steps, ensure that you have tightened the security of your system so this won't happen again right away, and then recover from the incident to bring your system back to it prior working state. Think of this as being like preparing for a multi-day hike in the mountains. (OK. Its really more like a forced march, but you don't have the option of staying at home. You have to do this, and do it right.) You don't just decide to do a multi-day hike and jump in your car. You need to know what to bring, what trails you will follow, how to use that little camp stove, and what kinds of obstacles you will encounter along the way. Please note that IT IS CRITICAL THAT YOU SPEND THE TIME TO KNOW HOW TO RESPOND APPROPRIATELY, OTHERWISE YOU WILL DO MORE HARM THAN GOOD. There is no "quick fix". Rushing and NOT READING and understanding the material referenced here usually means you get to do the job twice and prevent others from doing their jobs (by destroying evidence.) Another thing you must consider now is, is it worth it to leave the system connected to the Internet? Even though this system may be your department's web server, email server, etc., is it REALLY more important to stay online as it is to regain control? What use is it to spend days of work trying to clean up and recover (even with your web pages still available) only to have the attackers delete the entire file system, making you start all over again, because they continued to have complete access to your system via the network? At various points, like when re-installing and re-configuring the operating system before you go to the network to get patches (or if you need to leave for a while to get food or to sleep; you can't just do this for two straight days, after all), you might want to disconnect the system from the network so that your friends don't come back before you've finished your work. Where to begin? =============== Start right now by taking a few minutes to get an overview of the incident response cycle: http://staff.washington.edu/dittrich/talks/security/response.html The most critical things you need to do are: o Take good notes, including who did what, where, how, and when. Keep track of total time spent for all involved, for use in calculating damage estimates. o Notify the right people to get help. This may be any/all of the following: your own department's network administrators, security@cac.washington.edu, cert@cert.org, local/state/federal law enforcement. o Preserve evidence, in case the intrusion is more sophisticated than you think (and you have no way of knowing this, so assume the worst). If you fail to do this, you may ruin any chance of a complete resolution to this incident. At the first sign of a full system compromise, work to image the system (don't continue to dig around and especially DO NOT DELETE FILES you find.) o Enforce a "need to know" policy. Limit who is told about the incident, not so much to cover it up as to keep control of what is interpreted as "facts" from things that are really speculation or just plain false. Refer all questions up your management chain. o Use "out of band" communications. Sending reports from the compromised system, or from accounts that may have been stolen by the intruder, may let the intruder know (and react) to your efforts to contain them. o Contain the problem and eradicate it. This requires spending the time to learn the full scope of the intrusion, rather than just quickly cleaning up your own system. A sniffer may expose hundreds of other accounts and let the intruder "dig in" very deeply. Note that both lists recommend a couple of things; taking notes to remember what you have and haven't done yet (and how much time it took) and taking a snapshot of the system to ensure you preserve evidence in case it is needed. If you are not used to taking notes, now is a good time to learn that habit. Same thing for doing tape backups. If you don't have backups of your home directories and data files, the job of regaining control will be a bit harder, but you can still get the job done. When making backups for analysis, you might want to consider how they will be used. If you use a program like "ufsdump", which makes backups of file systems, it may not be possible for the person who is going to analyze the backup to read it, especially if they don't have the same operating system version as you are using. File system neutral backups (e.g., "tar", "cpio", etc.) can be read on other systems, but realize that these programs sometimes do not backup special files (e.g., device files, sockets, etc.) and they do not backup inactive (deleted) files. You may need to make two backups (one with "ufsdump" or "tar", another with "dd"), in order to both preserve ALL DATA on the system and to provide a snapshot of the files -- locations, time stamps, etc. -- that can be analyzed. Also consider the media. DLT, 8mm, 4mm, QIC tape, CD-ROM, ... These drives may not be at hand for those who will analyze the backup. When in doubt, ask what media type and format is required for analysis. (When sending data to security@cac.washington.edu, we can read 4mm (DDS-3) and 8mm tapes (DDS-1), and ISO 9660 format CD-ROMs, or raw "dd" image files transfered over the network as a last resort. We prefer GNU tar formatted archives for operating system independence when transering individual files from the system, but "dd" image copies are best for preservation of evidence -- including deleted files -- when analysis cannot be done immediately or where the possibility of prosecution exists.) Also, don't forget that if the attackers have full access to your system, they can read your mail and will know when you report the incident and what response you get. Do your communication from another system or by phone. Next, review one of these documentsg to learn about what you should look for in determining what has happened on your system, but don't start trying to fix things yet: http://www.cert.org/tech_tips/intruder_detection_checklist.html http://www.cert.org/tech_tips/win_intruder_detection_checklist.html In many cases, the intruders will have gained access to the root account (i.e., you have lost control of the system) and you may not be able to detect the full extent of what is happening on your system. Take a few more minutes and read the following document to understand what you are up against: ftp://info.cert.org/pub/cert_advisories/CA-94:01.network.monitoring.attacks If you are feeling absolutely compelled to do something at this point, and you can still get access to the root account, now would be a good time to do a complete backup. Use the "dd" program and make sure you get copies of all active partitions on the system. If you have never used "dd", see the examples in the following document, or contact security@cac.washington.edu for assistance: http://staff.washington.edu/dittrich/misc/forensics/ While the tape is being written, go get a cup of tea (coffee will compel you to take action again, which you don't want ;) and read the following document: http://www.cert.org/tech_tips/win-UNIX-system_compromise.html Also, you should know that the more skilled intruders have tools that allow them to hide their presence on your system, making the system lie to you. These tool kits are called "root kits", and they make life very hard for the novice administrator. Read more about root kits here: http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq More advanced methods of hiding programs and files on Unix systems exist as well, such writing programs that unlink themselves after they start, unlinking files (e.g., sniffer logs) after they are opened, and encrypting the contents of files and traffic that traverses the network. Dan Farmer and Weitse Venema gave a one-day course on computer forensics that covers these details. The course notes and examples they provide can help understand how to properly do forensic analysis and how to best preserve evidence. You can find PostScript versions of the notes, and tools used in the class, at: http://www.fish.com/security/forensics.html Remember that a skilled intruder can hide things from you, so don't assume that if you don't see a lot of signs of intrusion, your system was not compromised. ANYTHING strange things found on the system that would require system administrator priviledges to cause is enough to assume full system compromise and to start acting cautiously so as to not destroy what little evidence is still there. If you can't determine how someone broke in, you cannot prevent it happening again. Ready, set, go to work! ======================= Things will start to sound really familiar now. You will begin to get the picture about what you need to do. Use this better understanding to now build your plan of what steps you will start to take to deal with this incident. When CERT mentions security patches, figure out where these will come from and how you apply them. If you just re-install the operating system, but don't apply the necessary security patches, you're likely to have the attackers right back in your system again in no time. You DON'T want this. In some case, it is easiest just to re-install the operating system from scratch. Make sure you have good backups of your users' home directories and data files, since a re-installation may wipe out the entire hard drive and you'll have to put your home directories back on later. This is where that copy of the /etc/passwd file, /etc/group file and so forth, come in really handy. In other cases, you may be able to mount the original operating system CD-ROM and compare files by hand (make sure you use a program other than "sum", since the CERT advisory on "ongoing network monitoring attacks" that you read already told you that attackers modify this program to hide their "trojan horse" versions of programs like "ps" and "netstat". A good program to use is "md5": ftp://info.cert.org/pub/tools/md5/ Now start planning for setting up the system a little tighter than it was before. Take another few minutes and read the following: http://www.cert.org/tech_tips/unix_configuration_guidelines.html http://www.cert.org/tech_tips/win_configuration_guidelines.html Consider how you set up your accounts, how you choose your passwords, whether it really is a good idea to run sendmail on your workstation instead of using an email server on a centrally administered system, and what kinds of network services you leave enabled. This document refers you to a checklist for securing your system written by AUSCERT, the Australian CERT organization. C&C has also produced a similar checklist, although not quite as comprehensive and detailed as that of AUSCERT. Check them both out now: http://staff.washington.edu/dittrich/R870/security-checklist.html ftp://info.cert.org/pub/tech_tips/AUSCERT_checklist1.1 Now that you know what you need to do, make a short checklist and get your notes ready, then go ahead and get to work. You've got a lot of things to do, but you should have a pretty good picture in your mind of what is ahead of you. If you cannot even get into the root account to start, you will probably need to boot the system from a floppy or CD-ROM. If you have any questions about this, or need any other assistance, send email to security@cac.washington.edu.