Incident Response Procedures

Companyx.com Confidential

 

Overview

Computer security incidents are occurring at an ever-increasing rate on the Internet. Since we, Companyx.com, depend on the Internet for our lively hood, we must be prepared to respond to and investigate computer security incidents. This documents contains definitions of terms related to incident response, goals for the Companyx.com Incident Response Team, and procedures for handling reported security issues.
 
 

Definition of Terms

Event: An observable occurrence; an aspect of an investigation that can be documented, verified, and analyzed.

Incident: An adverse event or series of events that impact the security or ability to do business of Companyx.com or its subsidiaries.

Incident Severity Levels (map directly to your defined escalation levels):

1: incident could have long-term effects on business or incident affects critical systems.

2: incursion on non-critical system; detection of precursor to a focused attack; believed threat of an imminent attack.

3: threat of a future attack; detection of reconnaissance.

4: unsubstantiated rumor of security incident

Evidence: Data on which to base proof or to establish truth or falsehood

Forensic Analysis: Examination of material and/or data to determine its essential features and their relationship in an effort to discover evidence in a manner that is admissible in a court of law; post-mortem examination

Computer Incident Response Team: A group of technical investigators and security engineers that respond to and investigate security incidents

Computer Incident Response Management Team: A group that represents executive management, legal, public relations, and information systems management that oversees incident response and investigations
 

Goals

The goals of the Incident Response Team are to:

1. Maintain or restore business continuity
2. Defend against further attacks
3. Deter attackers through investigation and prosecution
4. Perform counter-intelligence/intelligence activities where appropriate

Procedures

A security incident begins when a security related event is reported to the companyx.com security team or operators. The following series of procedures should start as the event is being reported.

1.      Document Event(s)

Event documentation is a critical aspect of incident response and investigation. Without an accurate and verifiable trail of events, incident investigation becomes difficult and often ends without conclusion. Likewise, the response to the incident will not be complete if we cannot prove that the response will stop the adverse event(s) from occurring in the future.

Document the event thoroughly. DO NOT RUSH. Write down the following information:

• Date of event
• Time of event including time zone (as a relative offset from GMT, for example GMT-8 for Pacific Standard Time)
• Who or what reported the event. If a person reported the event include their full name, location, telephone number, and e-mail. If an automated system reported the event include the name of software package, name of the host where the software is installed, physical location of the host, host or CPU id of the host, network address of the host, and MAC address of the host if possible.
• A detailed description of the event. Include as much information as possible.
• Identification of the host(s) that the event is related to/occurred on. Include the hardware manufacturer, operating system type and version, name of the host, Companyx.com property number, physical location of the host, host or CPU id of the host, network address of the host, and MAC address of the host if possible. If the host has a modem connected to it, document the telephone number of the connected line or the wall jack number.
• If the event involves suspicious modifications or behavior of a computer that is accessible to many people and a person is reporting the incident, then ask the person for the names or descriptions of others in the area prior to and just after the event. If there is limited physical access to the computer, document the physical security controls that limit access (ask the person reporting the event to describe what they have to do to access the computer).

If a person is reporting the event, then instruct the person not to discuss the event with anyone other than the Incident Handler who is assigned to the incident, unless the incident handler instructs otherwise.
 
 

2.      Assign Event to an Incident

If the event appears to be related to an open or previously closed incident, then assign the event to the incident number (using an appropriate tracking system) and reactivate the incident if necessary. If the event appears to be a new incident, then assign an incident number and associate the event with the incident. 
 

3.      Assign Severity Level to Incident

If it is a new incident, or the new event related to an open incident is more severe than previous events, assign a severity level to the incident. Use good judgment, but err on the side of caution. If you think the severity is between two levels, then assign the incident the higher severity level. The following questions are intended to help classify the most serious risks, but are only meant as specific examples of applying the severity levels to security incidents:

Is confidential data at risk?

If there is imminent danger (the act is in progress) that confidential information can be read, modified, or destroyed by an unauthorized entity or the disclosure or access already occurred, then assign the incident severity level 1.

Is business continuity at risk?

If there is imminent danger of disruption of business due to security issues or malicious acts or the disruption is in progress, then assign the incident severity level 1.

Is public perception of the company at risk?

If there is imminent danger of modification of the public’s perception of the company due to security reasons other than disclosure of confidential information or disruption of service (i.e. main web page has been modified in an unauthorized manner, but orders can still be processed), then assign the incident severity level 2.

Someone identifies a security problem in companyx.com systems in a public forum

If the security problem could lead directly to the compromise of publicly accessible companyx.com hosts or customer information, then assign the incident severity level 2. Otherwise, assign the incident severity level 3 if there is no imminent threat to companyx.com systems or confidential data.

FOR SEVERITY 1 INCIDENTS, THE OWNER(S) OR OPERATOR(S) OF THE AFFECTED HOSTS SHOULD BE DIRECTED NOT TO USE OR MODIFY THE SYSTEM IN ANY WAY UNTIL THE INCIDENT HANDLER CONTACTS THEM AND INSTRUCTS THEM TO DO SO. For severity 2 incidents where unauthorized use has occurred, the same rule should be applied.
 
 

4.      Assign the Incident to a Handler

Assign the incident to an on-call Incident Handler (IH) and contact the IH immediately. If the Incident Handler cannot be reached or does not confirm that they are responding to the incident in the necessary time, then use the following escalation tables:

Severity Level 1-2
 

Contact	Escalate to	Escalation Time
Security-primary	Security-secondary	15 minutes
Security-secondary	Security-manager	30 minutes
Security-manager	CIO	60 minutes

 

Severity Level 3-5
 

Contact	Escalate to	Escalation Time
Security-primary	Security-secondary	8 hours
Security-secondary	Security-manager	8 hours

When an Incident Handler has been confirmed, contact the person who reported the event and give them the name and contact information for the assigned IH if it is a severity 1 or 2 incident.
 
 

5.      Coordinate Incident Response Team

The Incident Handler assigned to the incident is responsible for coordination of the response and investigation, and therefore will be the Primary Incident Handler (PIH) for the remainder of the investigation. The first task of the PIH is to review the incident documentation and associated event reports. The PIH should verify as much information as possible from the event report(s), verify the assigned severity level based on the available information, and acquire the resources necessary to respond to the incident.  The PIH should then go to the location of the incident if appropriate.

Law enforcement should be informed at management’s discretion. There are many factors to weigh including the severity of the incident, the scope of the compromise, cost to the company of supporting a criminal investigation, and the proprietary and confidential company information that would become public if a criminal investigation occurs. It will be up to the Computer Incident Response Management Team (CIRMT) to decide whether the incident warrants legal action. The PIH will be responsible for communicating the ongoing status of the response and investigation to the CIRMT. It is recommended that companyx.com legal counsel be present in all meetings with law enforcement relevant to ongoing investigations.
 
 

6.      Contain and Eradicate

The primary goal of the Incident Response Team is to maintain/restore business continuity, so containment of a security incident is vital. Containment and eradication methods are highly dependent on the type and scope of the security incident, therefore only sample scenarios and methods are provided. Ideally, the appropriate method can be extrapolated from the sample methods. The containment phase is also where the bulk of evidence will be preserved. Always keep the following in mind:

• Do not rush. Sacrifice time for thoroughness.
• Preserve as much evidence in its original form as possible.
• Take detailed notes on your actions and the actions of those around you, including the time of the action. Include your reason for taking the action. Sign and date the bottom of each page of notes.
• Record each piece of evidence you find, including a description, location, time found, and other distinguishing attributes. If it is physical evidence, record who handled the evidence before it came into your possession. If it is electronic evidence, record any processing of the evidence that occurred prior to your possession of the evidence. This data will help maintain an evidentiary chain and record of possible modification.
• If it is possible that a person who physically accessed the system caused the incident, preserve the physical evidence. Wear white cotton gloves when using the computer and handling physical evidence. Do not allow non-investigative personnel to enter the crime scene. Record the names and contact information of all people present when you entered the area. Make a record of all physical security controls in the area.
• Restrict information about the incident on a need to know basis. Only management and technical personnel (sysadmin, network-eng, development, etc.) that can significantly contribute to the resolution or investigation of the incident should be informed. Only disclose information that is immediately needed to solve the problem or task at hand.

If there are strong indications that the host has been compromised, then disconnect the host from the network. If possible, do not physically disconnect the host from the network; instead, assign the switch port the host is connected to an unassigned VLAN that does not provide network connectivity (this way link stays up on the hosts network interface, and a backup host can connected to the assigned VLAN for evidence gathering). There may be exceptions to this policy, but the exceptions should be few and well thought out. The longer an attacker has access to or control of a host, the higher the risk of long-term negative impact to the company.

 

Some forensic analysis of the running host needs to be done prior to shutting the host down to perform a disk copy.

For any computer system where an intrusion is suspected, there are several steps that must be taken prior to shutting down the system to collect evidence (see Evidence Collection Worksheet for further detail):

• Make a list of processes running on the system (i.e. “ps –auexww” on Unix and “pstat.exe” or “pview.exe” on NT).
• Check the status of the network interface(s) to see if they are in promiscuous mode (“ifconfig –a” and look for PROMISC flag on Unix and “querynic” on NT).
• Make a list of all listening ports and active network connections (“netstat –an”). If possible, make a list of processes that own the ports and network connections (“lsof” on Unix).
• Make copies of the executable files associated with every running process on the system or dump the contents of system memory to a file (??? How practical is this-probably not very on systems with large amounts of RAM)

It is important to remember that even mundane commands on a host can destroy valuable forensic evidence. Executing “ls” will change the access time of the current directory for example. Perform as few operations as possible that access or modify the file system prior to making a bit-for-bit copy of the file system on the compromised host. The preferred process is:

• Make two bit-for-bit copies of the compromised host’s hard drive
• Remove the original hard drive from the system and secure it as evidence
• Use one copy to aid the creation of a new system disk (copy only data that is known to be safe) then erase the disk using a secure wipe utility
• Use the second copy for forensics

Use only executables with verified integrity (“known good”) for these tasks to avoid Trojan Horses and modified binaries. Create an Incident Response CD-ROM that contains the appropriate binaries and documentation necessary for the system, but if not, use binaries from the original installation media. Be aware that many utilities rely on shared object libraries (libc-2.1.2.so etc.) that could be modified by an attacker, so sometimes just running “known good” copies of utilities may not be enough to protect from Trojan Horses. Clear any LD environment variables and then use the LD_LIBRARY_PATH environment variable on Unix systems to specify the use of integrity verified shared libraries (“man loader” for more information) and put integrity verified DLLs in the same directory as the executable used on Windows NT systems. The Incident Response CD-ROM should be configured to use these mechanisms, but the configuration should be manually verified before use.

In order to bring the host back online, the entrance point the attacker used to compromise the system must be determined. If the entrance point is clear, the scope of the breach minimal, and the integrity of the system provable, then it may be safe to patch the entrance point the attacker used and remove/restore any files they modified to bring the host back online. If the host can mount a NFS or SMB volume that is big enough to hold an image of the system hard drive, then make a binary copy (using “dd” for example) of the system drive on the remote volume so forensic analysis can be performed. Perform the copy prior to making the modifications necessary to bring the system back online.

In cases where the integrity of the system is not provable and/or the entrance point used is not clear, the system should be rebuilt from vendor-supplied media and backups, all known security patched should be applied, and unnecessary services should be disabled. Create a baseline for file integrity checks (Tripwire or equivalent), so the system can be closely monitored in case the entrance point has not been eradicated. Likewise, if the system has the CPU and disk resources necessary, enable system auditing for a period of time after the rebuilt system is deployed. Ideally, the forensic analysis will bring to light the entrance point used, and at that time the system can be positively secured against the attack and monitoring can be de-escalated to the normal level.

If there are signs of compromise on a system in a cluster, then all other member systems of the cluster should be immediately checked for similar signs. If the compromise is isolated to a subset of the systems, then the unaffected systems can be kept online, but they must be closely monitored until the entrance point is determined and a fix can be applied to the systems. If all the systems are compromised, then the standard containment and eradication procedures need to be applied to all systems in the cluster. If a system has a standby system to take its place, then the standby system should either have the entrance point fixed or be instrumented for aggressive monitoring before being brought online.
 
 

7.      Forensic Analysis

Forensic analysis will vary greatly from incident to incident, but the methodology should be consistent. The goal of forensic analysis is to discover evidence that proves:

• What happened
• Where it happened
• When it happened
• Who did it
• How they did it

In particular cases, forensic evidence will be used in criminal or civil legal cases against the perpetrator. Since it may not be apparent at the beginning of an incident investigation that the outcome will be a legal case, we must treat every investigation as if it will lead to a court case. Essentially, we must establish and maintain an evidentiary chain for all electronic and physical evidence collected during the investigation. We must also keep detailed logs of our actions and findings as investigators. Most computer crime investigations lack an evidentiary chain and detailed investigative logs, and that is a primary reason why it has been difficult to gain convictions in criminal cases. Also, be aware that this information will be available to the defense counsel through the information discovery process and may become public. Do not include company confidential information unless it is necessary.

To maintain an evidentiary chain the following information needs to be recorded:

• Where, when, and who discovered the evidence
• Who has handled or examined the evidence and when
• Who has had custody of the evidence, during what time period, and where it was stored/secured
• If the evidence has changed custody, how and when the transfer occurred (include shipping numbers etc.)

The relevant person should sign and date each entry. If the investigation leads to a court case, we must be able to prove that the evidence we discovered has been securely handled and not been tampered with.

 

All digital data analysis should be performed on trusted systems that can only be accessed by incident investigators.  Every precaution must be taken to not contaminate or co-mingle digital data from separate investigations.

It may be months or years before a case is brought to trail. As an incident investigator, you will be expected to recount your investigation in minute detail. Keeping a detailed log throughout an investigation will allow you to accurately recall all facets of the investigation. It is also important to document the investigation to establish a credible basis for any action the company may take as a result.

Document your hypothesis, how evidence supports or contradicts it, the actions you take to discover evidence or test the hypothesis, important or influential interactions with other people, relevant thoughts at the time, and anything else that will allow you to recall the investigation accurately. Include the time and date for each entry in your notes, and sign every page.
 

8.      Follow-up With External Organizations

In cases that involve attacks coming from the Internet, contacting Internet Service Providers (ISP) and other companies may be necessary to trace the source of the attack. Many ISPs and companies will only provide information if a subpoena is issued, but we have found that you will have better luck talking directly with their security staff managers than speaking with their helpdesk or customer support. Play up the "one security person to another" aspect-it will get you further.

If the source address of the attack is obviously spoofed, the ISP may or may not keep enough information to determine where the attacker?s traffic entered their network. In cases that involve high volumes of traffic from a spoofed source address, the chance that the ISP can determine where the traffic entered their network is much greater.

In most cases, the procedure will require contacting a series of ISPs and companies to determine the source (if possible at all). An important aspect to remember is to not disclose proprietary information in the process of talking to external companies, unless doing so is necessary and has management approval.

 

9.      Create an Executive Report and a Technical Report

After the incident has been contained, eradicated, and analyzed, create an executive level report (1-2 pages) for all severity 1-2 incidents that contains:

• A high-level description of the incident and its scope
• The impact on the company
• Actions taken to prevent further occurrences
• Recommendations for further action

The technical report should contain detailed information about the event, investigation, and conclusions.  All data used in the report should reference evidence collected and be verifiable.  The report should be suitable for use in court for either civil or criminal cases.

The reports will be given to the CIRMT and executive management for their review and response.

 

10.  Store Incident File and Evidence

All evidence, logs, and data associated with the incident should be placed in tamper resistant containers, grouped together, and put in limited access secure storage. Only incident investigators, executive management, and legal staff should have access to the storage facility. If and when evidence is turned over to law enforcement, create an itemized inventory of all the items, verify the inventory with the law enforcement representative, and have the representative sign and date the inventory list for our records. Give the original records to Legal, and save a copy for the security records. It is recommended that companyx.com legal counsel be present in all meetings with law enforcement relevant to ongoing investigations.