Workshop: Recovering From an Attack

Another attack earlier this year did more widespread damage and took longer to repair. About 300 Linux systems were hit by sophisticated and fast-moving attacks that used rootkits. Some 200 of these targeted systems were high-performance cluster machines, and it took the university more than a month to get one of the clusters back online.

We recommend watching traffic at the border so that you can keep abreast of any suspicious activity. Sometimes, other organizations can help pinpoint a problem. A sister university, for example, may alert us to a problem and send snippets of logs from machines under attack.

Recovery Mode

Once you've concluded that something is amiss, you can no longer trust any program installed on the infected machines, nor the operating system kernel. Assume every binary has been replaced and is either the problem itself or is no longer able to discover the source of the problem. Depending on your organization and the nature of the attack, you may want to call the authorities before attempting any reconnaissance or repair. (Don't forget to contact your legal counsel first.)

In the spring Linux incident at the university, a rootkit called SuckIT was surreptitiously installed to gain root access and steal user names and passwords, as well as to set off denial-of-service attacks using the unsuspecting machines. This attack is an example of how intruders have plenty of tools--including rootkits, key loggers and back doors--to choose from once they have access to your machine. It's difficult to determine whether the intruder has removed the software he or she installed or if it's still on the machine, so be suspicious of anything on that computer. With black-hat attacker tools like SuckIT and Wolff, intruders can burrow themselves and their code deep into your machines.