Network Security Blog

I’ve got to love it when a friend writes a post that disproves its own title.  For example, my friend Mike just wrote a blog post called “You are Not a Beautiful and Unique Snowflake” in which he goes own to explain that you’re unique, as our your competitors, but that doesn’t give you any reason to expect special treatment.  And that’s his real point, that while you may see someone else who’s in the same business, doing the same thing, there’s enough individuality and uniqueness that what appears like special treatment to you is really the outward symptom of the deeper differences between businesses that can only be seen during a thorough inspection of the real workings of the business.

Ask any experienced QSA about the different businesses they’ve worked with over their tenure and you’ll really start getting the feeling that there’s not that much variation in how companies do networking, configure servers and run a web site.  That’s only natural, since as human beings we generally tend to focus on how things are alike before we start observing the differences.  And from a casual viewpoint, there really aren’t any major differences between similar businesses when you’re taking that sort of 10,000 foot view.  But PCI isn’t about the 10,000 foot view, it’s about getting into the nitty gritty details of how credit cards flow through the businesses systems, where it’s stored and all the minutia of how every system that stores, processes or transmits cardholder data is configured.  If you consider how hard it is for even one business to configure all of their servers to a set of standards, then thinking about how much variation exists between any two companies, even ones doing the exact same business, should give you a moment of pause. 

Where Mike’s most dead on is when he says “You seem to think you know everything there is to know about your competitor, but in all likelihood you do not”.  I’m no longer surprised when I go into an assessment and somewhere halfway through a conversation a manager says, “Wait a minute, why haven’t I haven’t heard of this data repository/network connection/export to sales before now?”  It’s not a dig against anyone, the fact is most cardholder environments are complex and constantly changing and unless your only job is to dig into the environment on a daily basis, it’s very hard to keep up with what’s where.  Assuming you ever actually knew where everything is in the first place.  And if it’s not unusual to do this sort of accidental discovery in your own environment, how can anyone assume with any certainty that they understand their competitor’s environments well enough to make a judgment call on compliance?

It’s hard to remember sometimes how much of a difference a little segmentation or minor configuration changes with the exact same equipment configured just a little bit differently can make.  And part of the reason you consider someone your competitor is because they’re doing almost the same thing you’re doing, just a little differently.  Ask your own sales or marketing department how your product is different from your competitors and I’m willing to bet they could rattle off dozen differences in a couple of minutes.  (If they can’t, get a new sales/marketing department!)  If marketing knows that there are differences in the products, how can you reasonably expect that your cardholder data environment won’t have similar, nuanced variations?  The reality is, you can’t.

Do QSA’s miss things?  Yes, every day!  Are there QSA’s who ignore things they don’t want to review?  Probably, but that’s not an accusation anyone should make without proof.  Is you’re environment exactly the same as your competitors?  Unless a large part of your crew came from the competitor’s workforce, or vice versa, the chances are slim that when you actually look under the hood of how business is done you’ll find nearly as many similarities between you as you thought.  And it’s the ‘devil in the details’ that make all the difference in the world between passing an assessment and not.

Yes, you are beautiful and unique, just like everyone else.  And no, neither you nor your competitor are going to get special treatment under the PCI DSS.  They’re probably just not as similar as you thought they were.

I know it’s something I talk about at least once a year, feeling burnt out in my career path.  Like many people, I feel stressed by the huge amount of information that comes our way as security professionals, especially when I start reading about security breaches that potentially affect clients of mine.  It’s hard to feel like you’re winning the battle when you hear about a supposedly secure company that was compromised and had all their data exfiltrated.  It’s even worse when that data is probably going to be a lot of credit card information and there’s a ton of questions about the company’s last PCI assessment.

So there’s a little stress associated with being an assessor.  No matter how well you get along with the client, they know as well as you do that your job is to review their security configurations and setup, make a judgment call, and tell them whether they pass the assessment or not.  They didn’t want to deal with PCI in the first place, they don’t agree with many of the requirements, you’re the authority figure in charge of enforcing the requirements and almost invariably, you can’t meet with the time lines the client wants, usually due to circumstances beyond your control.  And who can blame them, since PCI is forcing so many merchants to spend time and money on security measures they didn’t want or didn’t feel they needed in the first place.  No one wants a third party telling them they have do put in AV or face having a higher exchange rate on each and every transaction.

It’s not an easy job, and while I am whining about it a little, what’s really surprised me lately is the number of QSA’s I know who’ve left the field in the last few months.  It’s not like people are telling me “I hate our company, I’m leaving for a better company”.  What I’m hearing is, “I hate PCI, I’m going back to some other aspect of security”.  For some it’s been the cyclic nature of PCI and going back to the same companies year after year and seeing the same exact issues show up each and every time. For others it’s been the lack of any significant changes in the PCI requirements since they came out and at least three more years before there’s much chance for change.  And in a few cases, it’s been the need to restrain themselves from commentary or criticism of PCI since it’s the main source of income.  Not a single one of the people I’ve talked to has said, “Oh, I’m sorry I left PCI, I want back in.”  And I don’t expect to hear that from anyone any time soon.

I don’t know if there’s a solution, other than training the next set of QSA’s. Companies are improving their security, some more than others, and it shows.  Unluckily, the bad guys appear to be able to bypass those security measures more adeptly than we are at putting them in place.  More security professionals are subscribing to the idea of “compliance through security, not security through compliance”, but it’s a slow process and too late to keep QSA burnout at bay.  Getting involved with the special interest groups (SIG) who work on many of the aspects of networking and security for PCI is a way to affect change, but it’s a slow process as well, and one fraught with it’s own perils and stresses.  You might be able to affect change eventually, but given the glacial pace that the SIGs have been releasing guidance, the chances are slim you’ll feel like you’re actually being effective at any point in the process.  And criticizing PCI, the PCI Council or any other aspect of the whole compliance effort is something that is always going to require careful thought and judgment no matter what role you take in the industry, something that’s not going to change.  Ever.   If you have a voice in the community, it can and will affect your job if you say the wrong things or say the right things in the wrong way.  Being right is no defense if you offend the wrong person in the compliance industry.  Not that I would know anything about that.

I suspect that we’re looking to see a spike in the burnout rate over the next year or so.  A lot of the people who have been involved exclusively in PCI since the early waves of compliance are reaching their pain threshold and looking for ways to get out.  Which is hard, since the skill set of a QSA (I know, oxymoron) is in high demand and pays well.  Even that hasn’t been enough to keep some people, since you eventually reach a point where the money isn’t worth it anymore.  People are going to continue to quit this segment of the industry, leaving holes that will have to be filled by less experience, though not necessarily less knowledgeable, assessors.   Which will in turn add to the stress most assessors are feeling.

I don’t personally have any plans in place to leave the PCI arena any time soon.  It’s hard, but fighting the stress, fighting the anxiety of being an assessor is something that can be dealt with for now, if not forever.  People I thought would never abandon PCI have left the field for other opportunities, so I know that I will also leave eventually.  And maybe this is just part of a natural career progression that occurs; learning a new skill, mastering it, then burning out and moving on.  But as opposed to leaving your company to go work in a different role elsewhere, most QSA’s leave one company to do the exact same thing at another company, until they burn out and leave entirely.  And I’m pretty sure that’s not a healthy career progression.

Martin and Rich are joined tonight by our new co-host, Joseph Sokoly, formerly of the Southern Fried Security podcast.  Martin leads off the night with a short story about his kids, in which he once again demonstrates his inability to remember the proper names for people and things (it’s Elevation of Privilege by Adam Shostack, not ‘escalation).  We talk about the most recent round of breach disclosures as well as a brief foray into PCI.  But we do keep it mercifully brief.  Welcome again to Mr. Sokoly, it’ll be nice to have someone a bit more reasonable on the show.

Network Security Podcast, Episode 235, March 29, 2011
Time:  28:08

Show Notes:

• PHPFog Blog:  How we got hacked and why it won’t happen again
• Netcraft:  Comodo hacked
• Hacker takes credit for attack on Comodo
• Romanian Duo hacks MySQL.com
• PCI compliant cloud providers:  No PCI panacea
• Tonight’s music:  The Fool You Took Me For by Bill Johnston

I always enjoy getting a chance to talk with folks like Gene Kim, Josh Corman and Mike Dahn.  We’ve talked about the nature of compliance together many times and I like that all of us have evolving opinions of how compliance influences the world.  We got the gang back together in front of a video camera to talk again about PCI and some of the things that have changed in the last six months.  Gene and I tell the story of how Mike and Josh started arguing last year and what we did to make them realize they were both saying much the same thing in different ways.  Rob Westervelt from SearchSecurity.com led Gene and I through the second part of the discussion, which was actually filmed before the first part of the discussion.  Don’t ask me how that works out, I think it’s one of those Hollywood effects things

RSA Conference 2011:  PCI Compliance:  Debating the benefits, unintended consequences Part 2, Gene Kim and Martin McKeay

RSA Conference 2011: PCI Compliance: Debating the benefits, unintended consequences Part 1 Mike Dahn and Josh Corman

PCI has become a way of life, not just for me personally, but nearly every merchant out there who makes a serious number of transactions using credit cards out there.  What was just a murmur on the wind for most folks five years ago has become part of the drumbeat of their every waking hour.  The evolution from something just the largest companies have to deal with to something nearly every merchant has to deal with has been painful for many, and will probably continue to be so for quite some time.  All because PCI has become the dominant standard in security for anyone dealing with credit cards.  And since most businesses deal with credit cards on some level, it’s become the de facto standard for the majority of the security industry.

Branden Williams wrote a post that took me back to some of the early arguments I had Josh Corman, very polarized ‘discussions’ of how PCI was the answer to a lot of security issues vs. PCI was deforming the security market in potentially harmful ways.  I’ve always been of the opinion that PCI is a good lever for getting money to implement projects most security teams should have in place.  Josh states that PCI has been starving other technologies that could go beyond the simple requirements involved in the standard, which is where his infamous “No Child Left Behind of Security” comes from.  Josh and I have since modified both our positions, I have admitted that there are some technologies that PCI ignores that could meet security concerns, while Josh has conceded that some of the technologies that PCI requires are something nearly every business needs in some form. 

I shouldn’t be surprised that there are still a number of merchants out there still in denial about what’s required of them by PCI and their acquiring banks, but I still am a little.  Many businesses have been able to ignore PCI or play lip service to it so far, but as the card brands push compliance (and the risk, incidentally) farther down into the merchant community, that’s going to be harder and harder.  If you’re a single merchant or service provider, no matter how large or small, you’re going to eventually have to concede that compliance is the irresistible force and you’re not an immovable object.  The only real ways to affect change on the PCI standards is going to be to work with them for now, work within the PCI Council and Special Interest Groups and potentially look for a better standard to be created.  Another alternative is to work with state and federal government to affect change, but I can’t imagine how that could improve the situation.

One last point from Brandon’s post to consider is the use of compensating controls; I’ve worked with more than a few merchants who were unable for real, technical reasons were unable to meet with requirements, as have most experienced QSA’s.  Don’t be afraid to use them, if you’re meeting and exceeding the requirement in question.  It’s not a blemish on your Report of Compliance to have a Compensating Control Worksheet, especially if it is well written.

If you’ve been anywhere near security during the last 18 months, you probably have a nervous twitch every time you hear someone mention the term Advanced Persistant Threat. Much like Cloud is the big term of this year’s RSA Conference, APT was last year’s buzzword. But that doesn’t mean that APT isn’t still a real issue and isn’t important; it just means marketing teams burnt out the industry on an important issue. Dave Merkel from Mandiant took a few minutes to talk to me about the panel he was on yesterday, as well as the PCI Council’s new PCI Forensics Investigator program. And yes, the two are more closely connected than is immediately obvious.

NSP Microcast, RSA 2011: Dave Merkel, Mandiant

There’s almost no one here at RSA who’s going to argue against the thought of Cloud being overhyped this year.  Every other talk seems to be about some aspect of the Cloud, even the talk I’m doing with Mike Dahn tomorrow.  But I seem to be a bit of a contrarian, saince I think that the hype is good, since it’s one of the few times that the security concerns around a technology are taking a bigger role than the push by the business to use the technology.  In other words, I believe the fact that it’s being overhyped now may mean we actually have a more secure solution over the next couple of years.  Or maybe I’m just naive.

David Spark caught up with me yesterday and asked about the hype behind the Cloud.  David had been told by a lot of people about the overhype, but I seem to be one of the few who see it as good.  Time will tell.

PCI compliance in the cloud is real, and Amazon is the first major cloud service provider to claim a PCI Compliant Cloud Solution.  Robert Zigweid, the Principal Compliance Consultant at IOActive, was the QSA who performed the assessment on Amazon’s environment.  We talk about what exactly Amazon is claiming is compliant in their PaaS solution, what a merchant should know this means and the difficulties of scoping and cutting up such a complex environment so you can assess it.

Network Security Podcast Microcast:  Robert Zigweid, IOActive – PCI Compliance in the cloud

Welcome to Day 2 of RSA

What would you ask for from the PCI Council this holiday season if you knew they couldn’t say no?  Other than abolishing the PCI requirements all together that is!   Walter Conway over at Storefront Backtalk already has his PCI wish list laid out, and he’s got some good ones in there.  Of course, Walt is trying to be mostly realistic with his wishes, listing a number of requests that actually might be attainable in the year 2011.  I on the other hand, feel no such compunction to be quite so restricted, since I don’t think even his fairly modest requests will actually be fulfilled.  And since we’re shooting for things that are improbable, it’s just as easy to shoot for the stars as the moon.

So what sort of things is Walt asking for?  First of all a list of all the training classes that the PCI Council will be offering in 2011.  Here it is two weeks before the end of the year and no one knows exactly when training will be available next year.  Personally that doesn’t mean much, since my training will be computer-based as a returning QSA, but for the people coming into the industry and the folks who want the newer Internal Security Assessor (ISA) certification, that’s a major issue.  There’s a major manpower issue in the PCI industry and if we can’t get more people trained, it’s only going to get worse.

The other very important item on Walt’s list is for all Level 2 merchants to get started on their own compliance for 2011.  If you’ve been involved in an assessment recently, you know that it’s a minimum of two months between when your QSA comes on site and when you have your Report on Compliance (RoC), and that’s only if you have everything buttoned down, there’s no remediation needed and the QSA company has a streamlined process.  Even then, two months is probably not realistic, you’re better off planning for three to six months, including remediation; add in the time to actually sign the contract with your QSA company and get a QSA assigned to you on top of that.  If you’re using an internal resource in your assessment, you need to look at the first request and realize that you may not even know when you’ll be able to get the required training this coming year.  Seriously, if you haven’t started planning for compliance in 2011 already, you better get started today, otherwise June 31 is going to sneak up on you and smack you on the back of the head before you know it.  And it will feel like you’ve been hit with a clue bat, trust me.

So what would I wish for from the industry and the PCI Council this Christmas if I knew they couldn’t turn me down?  Like I said in the beginning, I’d shoot for the stars; I want a complete rewrite of the PCI requirements that focuses on the desired outcomes, not the specific technical steps that need to be used to accomplish them.  Josh Corman had a good suggestion about this; keep the current requirements as an example of how to implement the new requirements, but we’d have a list that focuses more on the outcomes we want and less on the technology that is needed to make them happen.  The problem with this solution is that it would introduce a lot more wiggle room in DSS and would require a more mature, knowledgeable group of QSA’s, but it would also give merchants and service providers the ability to be more flexible in their solutions and maybe even allow them to concentrate on security first, compliance second.

And while we’re re-writing the PCI requirements, I want to drop the plethora of requirements that are redundant in any modern operating system.  We know that every modern OS tracks event type, time, user, etc., so why do we have to include that in every RoC?  If it’s there for applications that are developed internally and externally, then let’s make it apply to those and leave the redundant writing out of the process.  I’ve looked at most versions of Windows, Linux, Unix and mainframe, so I already know that they meet with the PCI requirements, so why do I have to write them up every single time?  No, this isn’t a point of frustration with me at all.

I don’t think Walt Conway and I are going to get anything other than coal in our stockings from PCI Santa this year, truth be told.  The process we’ve gone through the last few years indicates to me that PCI has calcified, which is quite frankly what almost everyone involved in PCI wants.  And when I say ‘almost everyone’ I mean the PCI Council, the majority of merchants, all the card brands and the acquiring banks.  Merchants have enough of a problem implementing many of the PCI requirements that effectively haven’t changed in over 5 years and won’t change for another 3.  The only people who really want change in the PCI standards are the security professionals who are charged with safeguarding your enterprise and the vendors who feel they were locked out of the market by PCI. 

By the way, there’s one more group who’s perfectly happy if the PCI standards don’t change and adapt:  the attackers.  Think on that for a little while.

Scoping is one of the most subjective parts of doing a PCI assessment.  What I consider to be a ‘connected system’ and what someone else considers to be the same can sometimes be substantially different.  The PCI Council has done a decent job of defining systems that “store, process or transmit” credit card data.  At least a good enough job that most QSA’s can agree in most instances what’s in scope and what’s not.  Whether they’ve done a good enough job that people who aren’t up to their elbows in PCI on a daily basis can understand scoping is a different question all together.

Well, Jeff Lowder has a short article on this that may be helpful (or at least give you an idea what your QSA is thinking), “How to define ‘connected systems’ in the PCI Cardholder Data Environment“.  One of the problems with scoping is that it’s changed gradually since the inception of PCI and rumor has it that there will be major changes from the Scoping Special Interest Group early next year. Jeff’s article is good in that he’s pointing out all of the connected systems you may have on your network which are currently being examined, but just for their supporting role.  In other words, your QSA may be checking out your AV server to verify updates, but it’s unlikely that he’s checking it out to the same depth that he’s checking your SQL database.  Which has a possibility to change with the updates to scoping guidance, requiring the level of scrutiny of both systems.   Think of your credit card data as an infectious agent that draws any system it touches into scope of the assessment, and you won’t be too far off target.