Hack and Counter-Hack Active Forensics: Tracking that Intruder. by Dragos Ruiu Jan 30 2001 Well this article is finally getting finished courtesy of the most excellent Palm folding keyboard (completely unsolicited user testimonial - err better make that ranting and raving) Which if you haven't seen it yet is the best little bit of engineering origami I have ever seen. And in reality, the 16Mhz Palm being roughly equivalent to the original Macintosh in capability (well 8Mb is a lot more than 128K Macs running off floppies but...), is just fine as a text entry device. But enough of that... the article title promised security arcana.... This article is about forensics.... which has probably been given a better reputation due to cheesy Hollywood exaggeration, but in turn has been glamorized to the point where one can forget how truly tedious it can be... without preparation. For, as we will discuss, a few preparations in this area can go a long way. Now I should also at this point outline my disclaimer - that these techniques go against some of the conventionally espoused "security expert" (one of the most dangerous boasts ever :-) techniques and procedures. But these techniques have worked for me and in my analysis present some of the best choices in my humble opinion to maintain data, computer, information process, and network, security. The computer security problems problems arise when the boundaries between these data security levels is improperly crossed (the common programming security mistake is forgetting that potentially all external input may be hostile) by unauthorized users - and usually the single biggest safeguard against these sorts of situations is the ability to go back to previous instances of the data before the access violation occurred - a trusted backup. Let me reiterate that for extreme emphasis: THE SINGLE BEST COMPUTER SECURITY TOOL IS A GOOD BACKUP STRATEGY. Sorry for shouting... but time and time again, the message is repeated to new sysadmins, only to have the importance of backups underestimated. And almost everyone (including me too I'm afraid), has their computers in a state of inadequate backups. And we all often forget that a good part of backups is redundant capacity. All too often, I see expensive external consultants from large firms hired for security audits advising customers to scrutinize small details of their software configurations while overlooking any analysis of the local backup strategy and redundant capacity. (Let's not talk about how many companies spend all their budgets on problem identification and then have none left over for remediation.) You can scan your systems with all the vulnerability scanners you like, but every system operator has to realize the basic fact that _no_ amount of effort will completely remove all vulnerabilities - there is always some risk. Ask anyone who does penetrations testing.... there is ALWAYS a way in... it's just a matter of time and resources - with enough chipping away, surely a vulnerability that will enable a wily hacker (;-) or more often an insider (:-(), compromise the security measures, A defender's objective is to make the workload of breaking in so hard as to make it generally implausible... and once the interlopers are in, to make detection and restoration of any modifications and or damage as low impact and speedy as possible. To this end a little bit of proactive preparations can go a long way - and in general, reactive security measures are far more expensive and complex than preliminary preparations. Detecting The Intruder The tricky part of all this is the detection phase... a competent system administrator will have a backup strategy that allows restoration of data to it's pristine, pre-intrusion state from the backups - once they know what has been modified. A big part of this detection work is the forensic analysis of the intrusion to establish exactly what vulnerability was used to get in and what was modified - because you can restore all you want and you will still have a problem if the door the hostile intruder used is still wide open and unlocked. Preparations for Detection A little bit of work up front can save many security hassles down the road. Some good measures to consider are: -Backups, I really can't emphasize this enough. One strategy that has worked well for me, particularly on mission critical systems is using removable SCSI or IDE carriers and buying disk drives in at least pairs... with an extra open carrier bay on servers not only is backup easy, as you can mirror the entire drive, it makes restoration and recovery lightning fast - and given the ever dramatically dropping price of spinning magnetic storage, a few extra disks this is a small price to pay for the dramatically powerful restoration and forensics capabilities such a strategy enables. The removable carriers add $50-100 to the cost of each disk, but let you image the disks and in case of problems be up again in the time duration of a single reboot - and it's a very smug satisfying feeling, playing with your intruders, who may have carefully invested a lot of time and work into stealthily installing their backdoors/trojans/rootkits/etc... only to have them magically disappear on the next reboot. Even if you haven't completed your forensics analysis and determined how they got in, this method allows rapid service restoration - and gives you another shot at logging the intrusion (if the intruder comes back, and they always seem to....) so that you can determine exactly what is going on. But if you haven't identified the entry vector, remember to make a second copy of your unmodified backup and use this backup to reboot from instead of using that precious backup - for the interloper may just go back in and re-hack it, and you'll be left without recourse. -The "Golden Master" - The above backup strategy also facilitates the next security preparatory measure I recommend: a "golden" reference master for your server installations. The most difficult part of incident response is often identifying exactly what modifications were made... and if you took the precaution of creating a reference system (disk image, etc...) to compare against then such forensics are simple and can be facilitated with a couple of small shell scripts and quick use of the unix checksum and diff utilities. Having a complete reference image will avoid many time-consuming laboriously intensive analysis sessions (and potentially very expensive expert consultant time) with other forensics toolkits such as Dan Farmer's and Wietse Venema's most exellent "The Coroner's Toolkit" [TCT] (found at www.porcupine.org) that you will have to resort to using for piecing together the fragments of whatever records may remain on the disk if you do not have such a master reference available Do not despair if you are in a reactive situation without such a master reference, because some of the traces that can be pulled off the disk with these TCT tools are frequently amazing - I've heard users of the TCT lazarus tool find old firewall configuration files on the disk that came from old operating system installs that span back to previous clean installs of different OSes from _two_ operating systems ago. There are some good articles on the use of these tools from the above authors at www.sysadminmag.com - but these are a laborious alternative, which can easily be avoided by the simple preparation of the "golden" reference master. Fingerprinting your system ------------------- Even if you do not have the capabilities to have enough redundant storage capacity to mirror your entire data-set you can get close to the same functionality by simply generating a file of cryptographic fingerprints for all your files and burning a set of these reference "signature" to a CD for later reference. To generate these signature data bases you can use commercial tools like tripwire, Enterasys' DragonSquire HIDS or open source alternatives like "rpm" or "nabou" to generate such signature databases, or (my favorite) the commonly found garden-variety unix/linux "find" and "md5sum" utilities can be used. The following command lines will scan through the disk and generate a catalogue of checksums/signatures: find / -type f \! -name "/proc"^ \! -name "/dev"^ -exec md5sum \{\} \; \ > checksumlist.master ls -laiR / > directory.master This command is what I often use to create a fingerprinting of a system after is is installed and before it is put into service - so that should any err... "unauthorized" modifications made by whatever interlopers are sneaking through your electronic neighborhood. Burn these files to cd or copy to a floppy and store in a safe (and remember that magnetic media has a maximum shelf life of about 8 years before corruption sets in if you're lucky). And you might as well copy over a few config files while you're at it too - it will make a system re-install, if it comes down to that, all that much simpler. Being fast at reinstalling fresh system is a good defensive strategy too. If you can rebuild a system with less work that the intruder has to take to Trojan it, you're winning in a fashion. Finding modifications: --------------- Let's start with detecting system modifications... or an area officially dubbed in research nomenclature, Host-based Intrusion Detection (HIDS). The goal is to detect unauthorized modifications to system and configuration files... A while back I wrote an article about network attacks, and in it I talked about the various phases of an attack: -reconaissance -penetration -embedding -control -data extraction -attack relay The steps following penetration rely on the ability to change the system, and have the changes stick (maintain control). This is the attacker's window of vulnerability - just like the penetration, data extraction, and attack relay can potentially be detected by Network Intrusion Detection Systems (NIDS) the modifications to embed the hacker tools and maintain control of the box offer the defender a chance to catch the interloper. While application data modifications can be fairly nefarious and notorious difficult to decypher, system modifications should, in a well prepared system environment, be straightforward to identify. The key however is in the preparation, for attempting this as a purely reactive measure is far more difficult. The work to prepare is not odious... you need to prepare a cryptographic fingerprint catalogue for all your systems and configurations and the other static files on your system. This fingerprint catalogue has to be backed up and securely stored for future reference. This is a minimum precaution... which will prepare against most intrusion systems (but emphatically not all). One recommendation I often make is to take this cryptographic fingerprint - a catalog "snaphot" of your system in it's "golden master" reference state - and burn several copies onto CDR... store one in an safe, and another for real exhaustive forensics (which you hope you never have to use, but may be necessary for insider intrusions) off site. It is amazing how much grief, work, and aggravation this relatively straightforward precaution can save. Further, more extensive precautions need to be taken care with the fingerprinting utility itself to protect against malicious modifications intended to hide other system changes... (For a good explanation of why, please read Ken Thompson's classic paper "On Trusting Trust" where he discusses trojaning compilers to produce "cooked" code even when the source code is clean) One good such precaution is to create a special auditing toolkit floppy or bootable CD (same thing really as the core of the bootable CD standard has a bootable floppy image at the beginning of the CD for the system BIOS to execute). I've volunteered for a while on one project that comes in handy for this, the tiny Trinux (www.trinux.org), security-oriented floppy Linux distro that is the masterpiece of Matthew Franz and company, but it is only my second favorite tool for this task. My favorite tool for this kind of forensic preparation is the SysAdmin tool box on a floppy that shoulod be in every good SysAdmin's pocket, tomsrtbt (www.toms.org). Billed as the most Linux you can fit on a single floppy it pulls the neat coup of formatting normal 1.4Mb floppies as 1.7Mb and replacing many normal unix commands and utilities with smaller more compact script based derivatives - it's truly amazing in these days of bloated operating systems just how much stuff is crammed into Tom's distro. Trinux does the same using the BusyBox toolkit (www.busybox.org). Using tomsrtbt ---------- On the tomsrtbt boot disk you'll find the find utility and the md5sum command which is all you really need to create these fingerprint catalogues. One other tool that deserves hobnorable mention in this category is the FreeBSD-based PicoBSD distro - though I keep waiting for someone to produce the equivalent mini-OpenBSD. You might want to create yourself a bootable CD with your fingerprint catalog and Trinux system as the boot catalog (I haven't successfully translated the funky tomsrtbt 1.7Mb image into the 1.4Mb or 2.8Mb boot images the CD boot standard requires yet, though it can be done and maybe some enterprising reader will tell me after reading this) so that you can perform forensics verification. So why all this focus on small bootable floppy systems? Well, one of the important concepts of computer forensics is that of a "trusted system." You certainly can't trust any utility on the compromised system - any of the binaries on it may have been trojaned and may make matters worse if you blindly go in there and start using the utilities on it. So you want to be able to import your own set of "trusted" tools into it for verification tasks... it doesn't require a lot of sophistication to replace the local verification tools (such as rpm, ls, md5sum, lsof, etc...) with special versions that hide the intruder's utilities and mods. So you want ot use your own. Discovering the Interlopers ------------------- How do you find/discover the intruders? Well there are a number of methods - but your first and most important line of defense is a SysAdmin (the human ;-) with a thorough knowledge of the internals and setup of the system. With careful network and system design the good Bastard Operator From Hell (and if you don't know what that is, go read this hilarious classic series right away, www...??? :-)can make an intruder's life miserable. Careful network segmentation and system design can go a long way. I like segmenting my traffic and isolating it by protocol type - so web servers only serve port 80 traffic and SMTP port 25 traffic is relegated to separate unique servers and so on (your boss may bitch and complain asking why you can't load them all onto one box - but explain to him/her that the cost of a server box is peanuts compared to the cost of downtime and each server is likely about the same cost as you will have to pay for one or two days of a forensics consultant's time in case of problems) Another way to detect the wily interloper is through special Intrusion Detection Systems(IDS), both host-based HIDS and network based NIDS (see www.snort.org for an excellent free NIDS tool -- well, I think so as one of the project volunteers ;-). In the above scenario, you can easily configure snort systems to function as burglar alarms and warn you of any non-port 25 traffic to the mail server and non port 80 traffic to the web server. Periodic, frequent review of system and IDS logs for unusual events and diagnostics is also important. Even with all the work and setup I've done on various IDS systems - I have to stress that a good understanding of your low level system design is crucial. I've caught more intruders (and been caught on pen tests :-) by hearing the sound of disk drives being accessed on servers that should have been idle than just about any defensive counter-measure I've ever rigged up. It is an unfortunate fact of life today that a firewall(the lock) is just not enough, as it will never be able to protect from exploits against vulnerable applications software, so any good commercial server farm ought to be wired up with an IDS (the burglar alarm). And why use just one... I use a mix of commercial and open-source IDS systems each tailored to look for specific kinds of intrusions. And I should mention that while commercial IDS'es are de rigeur for todays internet chaos, they are likely not enough. A common mantra of security professionals is "Defence in Depth" - meaning you should have multiple layers of defenses. And let me emphasize - just like any system can be hacked, _any_ IDS can be bypassed (some easier than others, but I'll save that for another article :-), so you should also try to keep some surprises up your sleeve. Good intruders know to look for IDS systems, and particularly know the weaknesses inherent in commercial products, so I always try to keep a few tricks up my sleeves to catch intruders off guard(seems only fair as they always seem to have a few tricks to catch me off guard). One trick I used to use all the time is to have a broken non-existent SMB remote disk mount on every system in a subdirectory off the root partition, which generates a diagnostic error message in syslog whenever it is touched... and I informed all my staff that they should never access this "tell-tale." However when the intruder stumbles in he won't know about this convention, and the first "ls -lR" he/she/it does will leave a noticeable error flag in the syslogs that may not look suspicious. BTW tricks such as these will also catch insiders in your SysAdmin team snooping about where they oughtn't... :-) (I used to use SMB because in those days the samba suite lacked a good utility to list disk mounts, further adding another layer of stealth and obfuscation to this booby trap.) Alarm Layout ---------- In general a good layout to ids systems should have multiple layers of alarms... I usually use a commercial IDS in the DMZ leading to the internet, and other proprietary systems (often based on snort, because it runs on a lot of platforms and it is fairly malleable) on the internal subsegments. Usually the alarming and response on the external net will be wired less rigorously on the outside than on the inside. It becomes tiring to have the big flashing lights activated and getting everybody up to Defcon-911 states every time some script kiddy(or stealth mode startups ;-) decides that scanning the entire internet would be a cool way to impress thier friends. The false alarms (the eternal achilles heel of any alarm system) quickly degrade into chicken-little syndrome, screaming about the sky falling with no-one listening - and even potentially becoming counter productive to increasing security, so some measure of restraint in the response to such external indicators ought to be exercised. Here lies the art of being a security officer. The internal IDSes should be wired more tightly and be monitored more closely. After all when they are going off you have some bigger problems - either someone has gotten through the front door of the firewall (and this is not so uncommon as you may think - for many organizations seem to have bitten on the false sense security pacifier of firewalls) or even worse, you have an insider rummaging about. There is another problem with the interior levels of defenses... though they should be checked more often and with greater frequency, the more finely tuned you have your system, the less false alarms and more monotony to this task - thus acting in opposition to the need for greater verification. Here is where patience becomes a good virtue for security officers. Ok, so now that you've hardened your systems to make intrusion more difficult, designed your architecture to highlight unauthorized use, and instrumented them with IDS burglar alarms you're set, right? Well almost... now it's time to plan for incident response. Ahh, here is where the dynamics and the tough job really begins... how to deal with the unwanted visitor.... Coping with Intruder Alerts ------------------ "Woop, Woop, Danger Will Robinson, someone has rooted your web-server" Ok, now what? Rule #1: check for a false alarm. Even with the increasing levels of mayhem on the publicly connected nets these days, this is still more often the case than not. As Mr. Douglas Adams so eloquently put it, "Don't Panic!" Network security isn't a thumb-candy twitch videogame (though I've often thought that it would make some good arcade game material...)... things happen much more slowly... sometimes agonizingly slowly. So take your time and verify your findings - check out whatever anomaly or unusual occurrence, alarm, or whatever has raised your suspicion, and verify that it isn't some natural occurrence. (And herein lies perhaps the most malicious activity of the OO-crazy programmers in Redmond, for they have accustomed the masses to computers that crash often and hard. I'm accustomed to uptimes in years on systems where I investigate every crash, so my daily crashing Windows systems are a security nightmare.) But, as you do this, once your suspicions have been aroused, do so with some discretion. Watch a little before doing anything noisy (like mailing out a global message to everyone in the company asking "Did any of you guys leave 'rootkit.tgz' on the server?") ...and here I'll diverge from conventional CISSP wisdom and say that punching power on the machine is about as noisy a signal/indicator to an intruder that they have been caught as the previous example. A good response if you can't leave the box alone to simply observe is to un-plug the network jack - after all networks are still lousy and users are very tolerant of abysmal service there. There are multiple advantages to keeping the box up even in a potentially infected state. The first is that you won't trigger any defensive response that your intruder may have planned for - buggering the shutdown and startup scripts is a common practice.... The other is that your forensics efforts may be greatly assisted by keeping the dynamic state of the OS and apps in RAM. Some intrusions are memory resident alone - particularly kernel modules, which seem to be coming into vogue again. One intruder had their rootkit wipe itself if it lost touch with the control system for longer than a certain period of time. There is much to be said for patience and watchfulness. And by not disconnecting the system and entering at the console or remotely yourself you can observe and learn a lot that can be crucial to your security. The only problem is that you are essentially on the same level as the interloper -- both of you have root. But you have time, physical console and media access and this should be enough to let you get the upper hand. But it may mean that you will have to resort to the same tricks as the hacker, having to import your defensive toolkit from other machines and removable media on to the machine, for any defensive software already on the machine may have already been discovered/compromised. With some preparation, though you can pop your "Hacker Surprise" handy tools disk in the server, copy to some surreptitious directory, change the dates to make it not stand out in a find, and bury your surprises like a needle in a haystack. I liked to use the more obscure sub-directories under etc or hidden in the man pages. With some work you can set up redundant logging, sniffing and all manners of dynamic tracing to record the hacker's activities. There's a whole other thesis on properly logging a machine, so for the sake orf retaining some form of brevity, I'll leave these steps to your creativity after outlining a few simple ones. Hack and Counter-Hack. History Lessons ------------ And now I get to dust off some old, old tricks.... For you see... a long time ago, on some computing architectures far, far, away from modernity, there used to be situations much like what happens when you have unwanted interlopers, and multiple priviledged operators are fighting for control of the same system... they used to be called time-sharing systems. Unlike today when administrators have the usual situation of being accustomed to having utter control of their processors, back when TTL chip dinosaurs roamed the earth, people used to cluster in terminal rooms all sharing one CPU. Much like what happens when you have an intruder that has acheived full administrator priviledges on your machine. In a romanticized view of this situation you can imagine high noon where you can battle for control. But as the maintainer of the system you ought to have home-turf advantage - and all your bunkers and emplacements built up and fortified in advance. By all rights, the deck should be stacked in your favor. But it's not so easy - for the intruder has several advantages: surprise, stealth, and difficulty of comprehension and location. You have to figure out what they're doing to your computer first. Ahh, a situation familiar to me from a long time ago. So let me digress into a story so we may hopefully learn form history instead of being doomed to repeat it: Learning to Hack ------------- When I started out becoming fascinated by computers (shortly after discovering Science Fiction, Asimov, Ellison, Star Wars, and that early crude x-wing simulation on an early IBM vector display at the University of Regina) the norm was to use them with paper printing terminals or punched cards (and even printed paper tape). Standard networking speed was 110 baud (or bits per second)... or roughly one tenth the rate of even a mediocre typist. The machines were slow and clunky, and rare. Home computers were a rare oddity consigned to home-brew wire-wrap monsters on s-100 cards (with the apple-2, Atari-800, TRS-80 and Commodore-Pet soon to change that). So if you were a computer enthusiast you would trudge over to the computer lab and line up for a terminal during the open lab times. But with this scarcity of CPU time resources came a thirst for more.... and thus came my first experiences with computer security - or the lack thereof. Because as a high-school student, I did not have the accounts and credentials to use these fantastic PDP machines and their troves of (mostly basic) games such as adventure, star trek (the old 8x8 printing game), eliza and others - the solution was to make yourself an account or find other means to gain access. But once accounts were accessed those rudimentary games and their primitive logic quickly paled in favor of the more challenging ones like cat and mouse with human opponents.... for there were others like me who were too young to get legitimate permission to utilize the computers, and in that throng of unruly young geeks the normal human competitive instincts surfaced, and the most popular game of all became a friendly rivalry to see who could get the most access to the most computer resources. For the first little while merely getting access to the various computer labs (crudely internetworked via serial lines strung between rooms) was challenge enough - but after basic access was established we young trouble makers quickly began to covet the administrative accounts that could control and allocated more CPU time usage (which was severely limited, as each student was given fixed allowances of this resource - and marks were even awarded for conservation of this resource in laboratory exercise assigned to students). As the rarest resource it quickly became the measure of ranking in the de-facto pecking order that was establishing itself amongst the hackers. The administrative accounts were quick to fall to login program trojans, social engineering, and even good old sneaking up to a terminal that the teaching assistants had left logged into priviledged accounts while they were busy elsewhere (lesson: there is no security at all if there is no physical security). But once the labs (which were dominated by IBM mainframes, and DEC PDP-8's and PDP-11's) were 0wn3d the real hacking games began. The game shifted from getting access to getting more and more computer time... and at the same time usurping your competitors tasks so that you could get progressively bigger and bigger pieces of the 0.5-10 Mhz CPUs that were shared by dozens of users simultaneously(!) - killing off your opponents tasks left more CPU for you and your games as well as being a game in itself. Once the number of youngster's sneaking in and achieving administrative access reached into the dozens factions began to emerge, and the individual efforts quickly segued into teams. And the power struggle for CPU time and accounts would ebb and flow. Roughly there were three teams - and when one team would get access to a new manual, learn a new trick or devise a method, they would be able to dominate these contests for periods of weeks at a time - until the next trick or countermeasure would obviate their advantage and another team's technique would become ascendant -- king of the hill, timesharing style, and mimicking today's constant exploit and countermeasure race. This back and forth of trick and countertrick continued on for months, each progressive generation of attack exploit becoming more automated, efficient, and powerful. The early ones were operator initiated at the keyboard (like when we first learned the existence of and how to bypass some of the security measures blocking access to systems calls that performed functions such as killing your opponent's processes) and usually initiated with small programs at the command line. Our primary battlefield was one particularly easily accessed PDP-11/34 running a now obscure operating system called RSTS/E - which had the most easily accessed terminal rooms (including that magic room of ancient teletypes that seemed to have been mostly forgotten by the TA's and stayed unlocked at all hours) and the scores of undergraduate labs that gave a much greater chance of blending into the flood of students un-noticed. As interlopers, we were all conscious that if one of us were to do something really bad or noticeably anti-social, the inevitable tightening of access controls would spoil the game for all, so the unwritten rule was that you could only affect the other hackers... the plodding freshmen struggling with their introductory computing assignments in BASIC were strictly off limits (well except for giving them assistance with their assignments in exchange for their login credentials and those all important CPU time balances). Later on the term wheel wars would be coined by academia for the kinds of games we indulged in, and you were only allowed to kill or affect others in the "game". This limitation persisted for a while, until the inevitable evolution of tricks and the discovery of the control and design of the process schedulers - and when the first group developed tricks to increase their attack script process priority to allow more CPU time and therefore more efficiency in killing off your opponent's processes - which led to a period of triumph for the most advanced team that developed this new technique, while the others puzzled over this new development and technique that gave such mysterious veracity to their attackers. By this point the original objective of these "games", to increase the run speed and the number of actual games you could play during your sessions at the computer, and the "game" had evolved into a far purer form where the only objective was to eradicate your opponents from the CPU (and thus I finally make my long winded way to the point of telling your these stories of ancient cyber-arcana in the first place). By these points, the competitiveness of the "game" had pushed us to access the computers at times other than the open lab times where the university maintained fairly open physical access to the terminal rooms, into picking and rigging locks on disused terminal rooms, sneaking in to class lectures in the terminal rooms and some pretty brazen unauthorized use of those computing resources - the TA's started to notice, and the university started to crack down. But by this point it was too late. The operating systems were riddled with trojans, back doors, surreptitious accounts we had created for ourselves utilizing many embedding techniques. The efforts of the computer science staff to rid themselves of the verminous little kids that infested their terminal rooms at odd hours were futile - for our knowledge and mastery of those computer systems far exceeded even the administrators and TA's - we had all the passwords to the priviledged accounts and had even created priviledged accounts that were hidden and invisible to them. All their efforts, password changes, access control methods were for naught - for by this point the hacker collective had managed to obtain full system documentation on os methods (for me finding out that my father's work library also contained full manual sets for the identical models of computers used at the university was a boon) and indeed our knowledge of the inner workings of the machines had reached the point of even rivaling the much more elite, well versed post-graduate research students. We grew cockyer and more brazen as successive measures of security and attempts to keep us out were foiled with great ease. After a while the TA's who were the alcolytes of these machines eventually grew frustrated and gave up on trying to keep us out - so they switched to co-opting us. If they couldn't get rid of us, then they could pawn off some of their dirty work on us - loading backup tapes, even assisting students in introductory courses with their assignments. At the time we weren't too concerned that we were being used to pawn off the menial maintenance tasks for the computers - the air of pseudo-legitimacy and frankly the rush of being able to play know-it all to people up to three times your ages was too tempting, and we gladly did their grunt work in exchange for almost legitimate access to the machines. Throughout this, the "game" continued, with the attack technology evolving in fits and spurts. No one group stayed on top for long - for with every new measure and technique came eventually some counter technique. The manual operator initiated processes yielded to increasingly sophisticated attack robots, that would spawn processes (monolithic and unifunctional at first, and then increasingly specialized for individual functions) Soon the humans at keyboards were outclassed and the operators had no chance of manually disposing of the processes without scripted or automated assistance. This was discovered the hard way one day, when one teams' attack processes got too good, and they began eating the cpu time noticeably so. The results were audible, as all the dot matrix printers in the room slowed down, increasingly until the individual characters being output by the central CPU slowed to the point where a sporadic "tick" from the printing terminals could be heard in one corner or the other could be heard every few seconds, and the confused freshmen started muttering and scratching their heads wondering what ailment struck their computer terminals. In the meanwhile two sets of attack programs were locked in a duel to the death, each spawning cloned copies of themselves and killing off the opponents processes as rapidly as it could allocate CPU priority... only by this point the systems had gotten too efficient and were exceeding the priority of the console tasks, and even the initiator of the programs no longer had control of them - uh oh. Now, imagine our panic - we weren't supposed to be there, and one of the major university computer labs was locked up to the point where it was unuseable.... and this was a machine that had a long complicated boot sequence, that involved loading a complex set of addresses on the busses using the front panel switches, not the kind of procedure to be done surreptitiously - so we had a problem. The problem was remedied shortly by the fire alarm.... and an important lesson was learned, which is my reason for recounting this whole story. I have seen the wheel of reincarnation follow these early pre-cambrian hacker patterns be rediscovered through the age of modems and bbses, then unsenet and uucp/fidonet networks, and now again in the internet era. The Net's Wheel of Reincarnation ------------------------- Just like in those early timesharing labs, network based intrusions seem to be following the same cycle, with crude operator driven exploits replaced by automatic attack scanners and worm rootkits... and similarly necessitating automated defences such as IDS and firewalls. The amount of information that needs to be processed to ensure system security is rapidly exceeding what may be done by a mere human, and is now necessitating program assistance. And in my opinion, active counter-hack systems will be the next evolution that will mimic that old timesharing evolutionary cycle. By using the same techniques as a hacker to enter a system and verify it, we will be able to catch even the newer more sophisticated rootkits. Much research work is now focused on profiling applications and execution, active protection against some exploit types like the Stackguard compiler to protect form buffer overflows, capabilities based systems, and other host based measures. In my opinion, we are just now entering the era of the automated attack robots with our internet technology... which will soon require autonomous defensive and forensics software (such as Intrusion Detection Systems) as the attack software becomes faster and more mechanized. Unfortunately, the main forensics tools we have now are still barely above the semi-automated systems in technology and automation: lsof - the handy utility that lists open files inzider - the windows equivalent of the above tool netstat - network statistics and connections dd - disk to disk copy or "Damn Dangerous" md5sum - fingerprint files find - search the disk for files ls - the good old directory lister ps - list active processes the details in the /proc filesystem The Foundstone utilities for windows based computers. Dan Farmer's and Wietse Venema's "The Coroner's Toolkit" with excellent utilities like the lazarus utility that brings back files from the dead - with remarkable efficacy. Another lesson that may be learned from the previous historical tale is that as soon as soon as a hacker countermeasure is discovered or developed, odds are high that another creative individual will find a way around it. There will always be some "0day" unreleased exploit that could evade all your measures. I'm sorry if I imply that constant paranoia is the only state a security professional should be in, but it's the sad truth. Let's walk through some forensics using today's tools. Passive Forensics: The Traditional Approach --------------------------------- First some checks that you should go through while the system is up (and potentially disconnected from the net): -Examine process tables and lists for unwated programs running. (ps ax and friend) -Check for Trojans and other unwanted programs that have network connections or socket bound to them using lsof or insider for Windows. You might also want ot keep your own trusted copy of the very handy lsof utility. -Use lsmod(or equivalent) and check running kernel modules. -Use top and other system monitors to look for errant system activity. -Look at device access times, temporary files and other indicators of system activity. Once you are satisfied no more information can be gained from memory state, shut the system down. A good precaution might also be to take a snapshot of /dev/kmem as root with: dd if=/dev/kmem of=/some/path/memorysnapshot I do not recommend going through an orderly shutdown. Under Unix if you have a lot of dynamic activity, use "sync" maybe, and then yank the power plug. It feels good too. J Then remove the media or boot the system with a trusted image from floppy or CD, or another disk. Try not to run the boot loader on the original disks if at all possible as this too may have been modified. dd piped into netcat can offer a convnient method of extracting disk images over the net. Take those checksum and directory commands and re-run them on the images. Use the diff utility to compare your original results and the new results. There are some system utilities like tripwire or the rpm verify command (rpm -Va) that can assist, but remember these too can be hacked. So use the same off system storage precautions as mentioned for the above built-ins. Verify the system install by checking system configuration files. A short list of top hack candidates is: /etc/inetd.conf or equivalent windows service startup configurations /etc/rc scripts and other system startup files like Startup folders Modifications in privileged system utility binaries and additions to the systems software Registry keys on Windows boxes, especially the startup services Password files and user lists to look for added user or elevated privileges on compromised accounts, don't forget accounts other than system logins such as database administrator password in the database softwar, passwords on remote control utilities and such, etc. Crontabs and other execution schedulers. Priviledged account .profiles, .cshrc, Startup Folders, and other games that count onhitting you when the administrator logs in. Key rings for authentication utilities, such as SSH, LDAP, Kerberos, etc. System libraries (DLLs, /lib) and kernel modules Look for obvious changes in modification times, inode numbers or other structural changes to the disk. Compare diligently against that "Golden Master." If you're very serious about finding stuff, look through the raw disk. See the TCT tools. Here an intimate understanding of the operations of your OS and server is a great facilitator. All that time spent learning about the software will pay off a great deal in this kind of analysis. This verification is best done from a "trusted system" such as Trinux, tomsrtbt, or PicoBSD as specified above. Also check out variations on your distribution install and repair floppies. I like tomsrtbt the most for these tasks because it will mount most kinds of disk partitions out of the box as well as giving net connectivity. Let me outline one very sophisticated intrusion I tracked down to highlight and emphasize the need for this kind of rigor when checking forensic data: We had one intruder that had successfully taken over many of our systems for months before they were discovered. They used a very nice trick that evaded all of our forensic measures, until the day we checked one server disk while booted from a "Trusted" tomsrtbt 1.7 Mb boot disk running in RAM disk: -They had trojaned the disk partition by hiding some special programs amongst the deleted, empty space and inodes on a disk partition. These files were essentially invisible, marked as empty/deleted, and the disk directory structure was corrupted to leave the space for these files still allocated. -They had replaced the system startup scripts with their own special versions and similarly replaced the system disk checking utility fsck (the equivalent of the Scandisk utility in Windows that verifies the system disk structure in case of an improper shutdown) with their own versions. -They moved the old startup scripts and original fsck to unused inodes, marked deleted at the end of the disk partition inode table into a spot not likely to be overwritten by regular disk activity and swapped in their Trojan rc scripts for booting initially. -Their rootkit involved the boot process. At system boot time the fsck utility would restore their trojaned rc scripts, which started in memory their backdoors with special obfuscation and hiding from the ps utilities and /proc filesystem. Then the scripts swapped their corrupted scripts and binaries (after they've run and were running in memory) with the original ones, and executed them as if nothing was amiss. The running system booted on the native partition would therefore look completely untouched to all dynamic system analysis. Even a copy of the filesystem to another system would not reveal the modifications for they were hidden except for a small window of time at boot. Only when we booted from another system on a floppy via tomsrtbt did their modified rc scripts become evident. So take this as a caveat; there are always cleverer techniques that may avoid even diligent forensics. The Next Level: Active Forensics ------------------------ The established wisdom today is to take a system down and do all the forensics post-mortem. I like a different approach, where you let the hackers keep accessing your systems, or even better set up special systems just to observe hackers (this concept is commonly called a HoneyPot, a name popularized by Lance Spitzner's ground breaking HoneyNet organization http://projects.honeynet.org). The problem with just taking the system off line and looking at the memory and disk images for forensics is that it can give you an incomplete picture of the intrusion. It probably will tell you how they are getting in and what they have done to your system, but... it may not let you establish: -Why are they breaking into your system? What is their goal or objective? -How skilled are they and how much of a threat do they really pose? -Who are they? Where are they coming from? Are you their target or merely a stepping stone or relay? What is the real target? -What knowledge do they have of your organization? -What has really been compromised? Mmmmm... Honey. ------------- A HoneyPot is a special system, non-critical, and usually it mirrors one of your critical active systems so the intruder cannot detect it being different from a production system. There are commercial HoneyPot systems such as ManTrap or The Deception Toolkit, but in my opinion a regular system that looks like your other systems but is instrumented carefully is a better HoneyPot, for even the commercial systems have telltale predictable characteristics that may be detected by a good hacker. And a server similar to your production systems will let you identify vulnerabilities used by hackers that also exist in your other systems as opposed to some synthetic simile. Typically the HoneyPot is configured to be less secured that your other systems - and this simple step is usually enough to make sure it is penetrated first, giving you a tell-tale alarm of problems. I've been running HoneyPot systems for several years, and frankly I'm disappointed with the lack of creativity evidenced by most hackers. They always go for the bait/candy and the path of least resistance first - the easy vulnerable system. Inevitably, when faced with several systems, they attack the most insecure one -- usually the HoneyPot. There is a fine line to be walked with a HoneyPot, for it may be harder to prosecute a penetration on a system that is designed to be "inviting" to hack -- but in my opinion the additional hacker intel that is gained from these systems is worth it. One recommended setup for a HoneyPot is a second "watcher" system, running Snort in sniffer mode, or some other data logger, logging all traffic into and out of the HoneyPot (as it is all suspect) to disk. An advanced honeypot will also send traffic to and from the network, making this task more complex, but a little creativity here will go a long way. Counter-Hack! ------------ Once they are in, and the alarms are sounding, I like to go back in at the console and start watching. If you want to be very careful, you can make your watcher tools use the same techniques as the hacking tools to hide and obscure themselves, hide from top and ps, stealth kernel modules, and the rest of the tricks. That way they can't see you watching them. Some of my favorite tricks are shell script one liners that I learned back in those time-sharing days. These have the advantages of being highly portable (type them in from memory), and don't require the storage of any tell tale binaries and their function very difficult to decypher from operational monitors like ps. Here is a cute one: touch last ; while (true) do nice -n 19 lsof -n | grep -v "^lsof" | grep -v "^mv" | grep -v "^grep" | grep -v "^diff" > latest ; diff last latest | grep -v "^[<>]" | tee - funnyname-stealthlog ; mv - latest last ; echo -n [$SEQ += 1] | tee -a funnyname-stealthlog ; date | tee -a funnyname-stealthlog ; done (If you are not using bash and your sh pukes on the echo and $SEQ math just omit this step that prints unique sequence numbers to verify against edits in the stealth log files.) Let's walk through that one. This is a one-liner that runs lsof (which lists the open files on the system) at a ridiculously high priority. Sure it'll shoot your performance to crap, but I find few systems these days that are CPU bound except some IDSes and DB servers, and one time an intruder hacked into my system and started running a dictionary/brute force attack on my own password password file using my own spare CPU cycles on a web server. Cheeky bugger. I just feel more comfortable in a defensive situation knowing that I'm getting as many of the CPU cycles as I can. More for me is less for the unauthorized intruder. So nice -n 19, eat that scheduler. This will also make the next part run at better granularity and miss the least amount of file opens and closes because it will run more often than the interloper typically. The next part of the command pipes the lsof output through a set of filters for active files, which I did using grep for some obfuscation but could have been done through sed or a myriad of other methods that will be available as binaries on almost all systems. If you find certain programs too noisy, just add some more filter steps in there. I log all the changes in the lsof output between the current one and the last one, which correspond to file opens and closes on the system to the screen and a log file on disk after adding a timestamp, and a sequence number to detect any funny roll back the file pointers to a snapshot, log-hiding tricks, or editing - in case your obscure surprise log is discovered by the intruders. Further variations with netcat and other contortions are possible as small, simple, enhancements. Play with it -- and if you become familiar with the typical file activity and signature on your systems before they are hacked, this little tool will give you an amazingly cool trace of any files the attacker's programs touched on the disk. In similar veins I like to do this too: touch oldsums; while (true) do nice find / -type f \! -name "/proc" \ \! -name "/dev" -exec md5sum \{\} \; > sums ; diff sums oldsums | grep -v "^[<>]" | tee -a slowlog ; mv -f sums oldsums ; \ echo -n [$SEQ += 1] | tee -a slowlog ; date | tee -a slowlog ; done ...and... touch oldsums; while (true) do ls -laiR / > dir ; diff dir olddir | grep -v "^[<>]" | tee -a permlog ; mv -f dir olddir ; \ echo -n [$SEQ += 1] | tee -a permlog ; date | tee -a permlog ; done There are many variations. Invent your own and confuse a hacker. :-) Try playing with the above on some systems to get a feel for them. But one caveat during hostile situation use: watch-out for the system console and key loggers. Key loggers seem to be coming back into vogue. A savvy intruder will be able to foil these tricks above if they figure out your particular surprise variation. One time when several hackers kept coming in at the wee hours of the morning, I toyed with them a little, after grumbling because of the alarm set up to wake me, or having to stay up. I started typing in the first of the above loggers. Very slowly. Excruciatingly slowly (hopefully, annoyingly slowly, because I was sure annoyed at the bother :-( ). I suspected keyboard logging and wanted to test out my hypothesis. So I typed at snail paces to see at which point they would act -- and you could almost see them puzzling at the other end of some obscure internet link scratching their heads and wondering "What the hell is he typing? What kinds of messed up command is that?" And just when I started past the mv command (typing at about a letter every ten seconds) and the intent of the command became clear... boom, my box did a hard reboot.... Well I guess that showed me they were listening... Time to look for the key logger. But, key loggers can be useful for the defender too. In general, every offensive method can also have a defensive application and vice versa. - they're all two edged swords. The above are only a few methods of adding unexpected logging. A while back I posted to bugtraq a network sniffer that can be added to a Linux kernel and runs inside the kernel, dumping packets, while still at less than 100 lines being small enough that it or a variation can be typed in from memory (It was called the klog patch)... There are many more possible approaches to Active Forensics and setting up HoneyPots. Conclusion These are just a few simple tricks you can use to find out if and how your systems are being penetrated, and some precautions that can make a security officer's life much, much, easier in case the unwanted occurs. There are endless variations to these tricks and preparations, and I hope this article will inspire you to develop your own site-specific and unique defensive tools. The concept of defensive "0days" is just as valid as exploit "0days", so I hope these examples can stir new creative tools (and if you come up with some cool ones, please send them to me if you are willing to share, for I might have some cool new ones too ;-). In computer security and network security if you can adopt the hacker's toolkit and methods to supplement your own, you will be that much more a formidable defender in my opinion. To paraphrase those perl guys, "There's more than one way to hack a hacker!" and to borrow from Monty Python, "Nobody Expects the...." Surprise HoneyPot or Active Forensic Logger. Oh, yeah... and one more thing again.... have you checked your backups? :-) cheers, --dr ------------------------------------------------------- -- Dragos Ruiu dursec.com ltd. / kyx.net - we're from the future gpg/pgp key on file at wwwkeys.pgp.net or at http://dursec.com/drkey.asc http://cansecwest.com CanSecWest/core01: March 28-30, Vancouver B.C. ------------^ Speakers: Renaud Deraison/Nessus Attack Scanner, Martin Roesch/Snort/Advanced IDS, Ron Gula/Enterasys/IDS Evasion, Dug Song/Arbor Networks/Monkey in the Middle, RFP/Whisker2.0 and other fun, Mixter/2XS/Distributed Apps, Theo DeRaadt/OpenBSD, K2/w00w00/ADMutate, HD Moore/Digital Defense/Making NT Bleed, Frank Heidt/@Stake, Matthew Franz/Cisco/Trinux/Security Models, Fyodor/insecure.org/Network Mapping, Lance Spitzner/Sun/Honeynet Fun, Robert Graham/NetworkICE/IDS Technology Demo, Kurt Seifried/SecurityPortal/Crypto: 2-Edged Sword, Dave Dittrich/UW/Forensics, Sebastien Lacoste-Seris & Nicolas Fischbach/COLT Telecom/Securite.Org/Kerberized SSH Deployment, Jay Beale/MandrakeSoft/Bastille-Linux/Securing Linux