May 21 2010

Are low standards better than no standards?

Published by at 9:18 am under Hacking,PCI,Simple Security

On Twitter this morning, @secrunner made the following comment:

“I think it’s surprising that PCI still hasn’t developed a program to certify pen testers or at least standardize the approach”

In reply I stated that given the level of certification for ASV’s (Approved Scanning Vendors), I’m just as happy if the PCI Council would stay out of the business of certifying pen testers or creating a standardized approach.  In reply @secrunner asked the following:

“In the spirit of PCI, isn’t even some standard (even a low one) better than none?”

The answer is, no, it’s not.  To be more specific, my answer was “Low standards for a merchant are better than nothing.  Low standards for a vendor are misleading at best, dangerous at worst.”  Let me explain why I think this way:

When you go shopping, one of the last things on your mind is probably “How does this merchant protect my cardholder information?”.  It’s one of the first things I think of, but that’s what I do for a living.  Most people are just concerned about if their merchant is going to have their size or the best price on the new toy they want.  They just assume the merchant has taken the necessary steps to secure their cardholder information.  And if they haven’t, consumers know that they’re only responsible for the first $50 dollars worth of fraud, and even that is usually absorbed by their bank or credit card company.  Sure, getting a new card issued to you is a bit of a hassle, but for most people it’s something that’s over and done with in a few minutes.

In this case, security is assumed and is not the primary concern of the person doing the purchasing.  A default standard such as the Payment Card Industry (PCI) Data Security Standards (DSS) is important and useful because it gives a baseline level of security for the industry to meet.  It may not be the level of security the company really needs to protect themselves, but all too often this baseline is more than the company was doing prior to the standard.  It may not be perfect security, but at least it pulls you up from the level of ‘low hanging fruit’. 

Certifying a vendor as a ‘compliant’ or ‘certified’ is a completely different story.  When an industry group such as the PCI Council makes a standard for a class of vendor and then certifies these vendors as meeting a certain baseline, this certification becomes one of the primary influencers in the purchasing decision.  Using the ASV certification as an example, a merchant won’t even consider a scanning vendor for their company unless the PCI Council has already certified them.  The merchant has to use a vendor who’s been certified otherwise they can’t submit the scans as part of their own compliance.  A large part of why this works is that external scanning of web sites is a fairly well understood, repeatable and, most importantly, testable process.  It can be easily automated and running the same test against the same site ten times will generally generate the same results every time (okay, maybe 90% of the time)

Penetration testing is an entirely different issue.  Yes, there are automated tools.  Yes, some pen testers don’t go much beyond that level.  But the good pen testers I know treat penetrating a company’s defenses more like an art than a science.  Metasploit and other tools are their paintbrushes, but it’s the person who’s using the tools that is actually making it possible to find the vulnerabilities in your company so that you can shore up your weaknesses and prevent someone else from finding them.  This isn’t a process that easily documented, standardized or testable.  It might be something you can do on a person by person basis, just as the PCI Council does for QSA’s now, but it would be nearly impossible to do for a company.

Let’s be honest, in the PCI-DSS, the idea of ‘penetration test’ is barely even defined.  It has to have a network portion and an application portion, but the how’s and what’s of penetration testing are left up to the QSA to verify and validate.  There’s no agreed upon standard in the industry of what makes a pen test a valid and acceptable pen test, let alone within the PCI community.  If the PCI Council wanted to certify pen testing companies, the first major hurdle they’d run into is making up that definition.  Then they’d have to come up with a way of testing companies’ adherence to the standards and create a certification program.  This would be a huge battle to undertake and the benefits would be minimal. 

Right now, it’s up to market pressures and QSA’s to determine what’s a ‘real’ penetration test.  If someone created a penetration testing certification there’s only one group of people it’d help:  marketing.  Most merchants wouldn’t read the requirements for the certification, they’d just use the certification process as a check box to weed out potential vendors.  And I can guarantee that the marketing teams would love that.  And I doubt it would make the results of penetration tests any better; in my opinion it would simply mean that most companies would ‘dumb down’ whatever they’re currently doing so that it met with the minimum standards and no more.  I much prefer seeing the merchant who’s having the pen test performed ask questions about exactly what’s going to be done and try to understand what they’re getting themselves in for.

[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

3 responses so far

3 Responses to “Are low standards better than no standards?”

  1. Diami03on 21 May 2010 at 9:24 am

    A+ for how you broke down DSS staying away from pen testers!!!!

  2. Eric Irvinon 21 May 2010 at 7:32 pm

    Well delivered. I think this would make a great round-table discussion at some point. While you’ve made some good points, there are still questions such as, why even require a pen test, if the quality of such a test doesn’t matter. The same should go for the ASV, why even require the cert if the results don’t matter. While I agree that a checkmark from an ASV, just like a Pen Tester, and for that matter, a QSA lulls many in to a false sense of security (literally), we have to recall the intent of the requirements.

    At the same time, I also acknowledge that as you said, if a vendor chooses to use the qualification bar, as a ceiling instead of the floor, then we are truly all providing our customers a disservice.

  3. […] Are Low Standards Better Than No Standards? Nope. […]

%d bloggers like this: