Nov
10
2008
Michael Chertoff, the Secretary of the Department of Homeland Security, will be here in California tomorrow. He’s hosting a blogger roundtable on Cybersecurity and I’m one of an unknown number of security bloggers who’ll be attending the event and talking to Mr. Chertoff face to face. Quite frankly I was surprised that the Department of Homeland Security was even aware of blogs, let alone willing to step out of Washington to talk to us in person. I probably shouldn’t be, since the TSA has had a blog for months now, even if I rarely agree with what they post there and never take it at face value.
Mr. Chertoff is on his way out due to the change in leadership our country is going through, but he’s held a highly political and thankless job for some time now. He has a unique view of the security of not only our nation, but every nation in the world. So what would you ask the man who’s been responsible for ‘homeland security’? What do you want to know about how we’re doing security at the highest levels? What burning questions about the TSA and your shoes are eating away at you? If it was you going to talk to Mr. Chertoff tomorrow, what’s the one question you’ld ask?
I have a number of my own questions, but I know that you can come up with even better. Leave a comment on this post with the question you’d ask. Keep it short and concise, make it topical to cybersecurity. I won’t be asking any ‘attack’ questions, but I’m perfectly willing to ask some of the hard questions. Personally, I want to know what it’s like to be placed in charge of Homeland Security without any real power to affect change? Except that most security managers already know what that’s like.
We’re allowed to bring cameras and audio equipment, but no video. Most of my equipment is for close up interviews, but I’ll do the best I can with what I have. I’m just hoping the Secret Service doesn’t decide that some of my equipment isn’t acceptable. Or decide that I’m a security risk at the last minute.
Nov
10
2008
Congratulations to Jason, the winner of the free pass to CSI. Here’s his story about how a minor change to a script almost caused a major disaster. I have my own war story about scripts I’ll share later this week. Here’s a hint: Always make sure you’re in the proper directory when running your scripts.
This happened when I was first learning to admin UNIX boxes. Another
SysAdmin and I were working on a shell script to lowercase the file
names of 30-40 million image files. They were on an NFS mount that was
used by several servers. These images were part of detail listings of a
relatively busy web site and we were right in the middle of the day.
Now that the background of the mess are fully explained, the story
gets going. We went through several revisions and were testing against
a directory on a desktop system. Nothing destructive happened during
testing and we were getting fairly comfortable with the “safety” of the
script.
We finally thought we had a working script, so we moved it to the
prod server. Then we noticed a “minor” change that needed to be made on
it. We made the change then decided that since this was a such a small,
little tweak we could run it on the live NFS mount without any further
testing. Fire in the hole!
The script took off and we watched it run. All was well. Then my
phone rang from the NOC. A panicked operator was on the phone saying,
“Hey what’s happening with listing images from xyz.com? They are all
coming up as 404s!” I killed the script while thinking some thing like
“oh crap, oh crap, oh crap!” Sure enough the script had wiped out about
50% of the images. Amazing how fast a shell script can delete when it
goes haywire.
We pointed the web servers to a backup copy of the images, then
started to recover to the production mount. The backup was a couple
days old, so our image processing guys had to re-upload the missing
work. I was lucky that the online backup was there. I had taken it for
reasons unrelated to this event. The next day I got to explain to the
CIO what had happened.
The moral of the story was backup first and test your script until
it is golden before going live. Then test it again and again and again.
Make sure you are doing at the proper time, then go to production. We
didn’t have change control, so I’d add get all the approvals now too.
Cover your butt.
It was a good lesson. I’ve never done anything like that again in the last 7 years.