Dec 20 2003
Incident Response
I’m always have to be cautious about what I put on this site. I know at least one of my co-workers read the site from time to time, and I don’t want to disclose any secrets of the internal workings of the business I work for. On the other hand, I want to let others know about some of the trials and tribulations I have to go through on a daily basis, so they can take heed of my mistakes. I’m sure that someone will let me know when I step over the line.
The last two days have been very exciting at work, and not in a positive way. You would think that, by now, every server in the corporation would be patched (with the major ones, at least), have anti-virus, and be properly managed. But that is obviously too much to ask. On Thursday afternoon, I came back from lunch to find that we had an infestation of the Nachi worm. It turns out that a host that had become infected several months ago had been unplugged and placed in a closet. When it was brought out Thursday morning, the infection began.
As Nachi is want to do, the systems began by sending pings to an external subnet and when they were done with that, they began pinging the internal subnets. Thanks to the configuration of the firewalls, none of the pings to the external sites were successful, so the next stage of the virus wasn’t invoked. The internal hosts weren’t quite so lucky. By the time we controlled the incident, ten seperate hosts had been comprimised.
I know ten doesn’t sound like a very large number, but these were at our main office, and the amount of traffic they sent out caused a noticable lag on our network. Additionally, four of the systems infected were part of the incoming check flow process, and caused the checks to be deposited a day late. Again, this doesn’t sound like much, but the interest on one day’s worth of checks can amount to more than I make in an entire month! All told, this incursion probably cost approximately 20 hours of work for our network group, and at least 40 hours for the security teams time.
Our first response was to call all the interested parties in the network and security groups, then start shutting down the switch ports that infected systems were on. This was mostly effective, though in one case, the user simply moved the computer to another port. It made hunting down the systems a little harder, but cutting them off protected the rest of the network. Our security group then ferreted out the systems and began cleaning up the damage. In at least one case, the infected system is being re-imaged. Most of the systems are back up with the proper anti-virus protection, but there are several systems that are actually owned by a third-party vendor and will have to be managed with their cooperation.
I have two main criticisms with the way our organization handled the infection. One, our incident response was created ad hoc. We have no policies or procedures for the handling of such incidents, though I am going to be hosting a forum next month that will hopefully lead to the creation of a Computer Incident Response Team (CIRT). My second issue is with the way the security group handled their part of the incident. They do not want to be responsible for clean-up in these situations, they do not want to have any disciplinary action, they do not want to call attention to themselves. In my opinion, this is exactly what a security team is there to do!
Even though they are responsible for virus control, security would prefer if the local system administrators handle the day-to-day clean-up. I can understand this, since there is very limited manpower in the security group. Security can’t impose any sort of disciplinary action, because their just published corporate policies contain absolutely no consequences for breaking any of the policies. This had been pointed out before the policies were published, but it was the decision of Upper Management to not include consequences. So much for managment support and buy-in. And the desire to be overlooked or viewed as a ‘kinder, gentler’ security department comes from an inability to perform any effective change in the corporation.
My first issue is hopefully in the process of being addressed, but my problem with our security team may be too deeply ingrained in the culture of the corporation to be fixed. The IT department, and security in particular, has historically been an obstacle to overcome, not an ally to court, in this company. Rather than being a group of enablers, the IT department has often been seen as one of the main hurdles to outside groups. This isn’t an unusual view of IT everywhere, but it is exaggerated in the current environment.
I know I’m being hard on the security group, but as a CISSP, I am supposed to have a vested interest in the security of the corporation. Watching a group such as this refuse to even try to change the environment they work in is very frustrating to me. I work for a different group, with a different manager, and I have to do the best I can, either with the security groups cooperation, or despite them. God help me, but I too am beginning to view the security group as an obstacle to overcome.