Network Security Blog

I really wish I had the time to fully explore the idea, but there’s a certain amount of resonance between the criticisms Adrian Lane at Securosis levels against Visa’s guidance on  tokenization and criticism of the PCI security standards in general.  I believe we’re to the stage as an industry that we mainly agree that the PCI standards are a good starting point but there’s so much more the PCI Council could be requiring merchants and service providers to do for security.  Visa’s guidance is much the same way, it’s a good start, but it could have been so much more.  And in both cases, I believe the reasons for the compromises can be boiled down to not wanting to require too much of the community and not wanting to limit the flexibility of the standards too much.

I believe that the Visa best practice papers for tokenization and truncation are just like the PCI standards themselves; they’re a good place to start your journey, but these requirements aren’t enough to build your entire security stance from.  It’s up to you to continue from here to determine how the particular technologies are going to impact and secure your environment.  I think the difference between providing guidance and issuing edicts is something we’ll be talking about next Sunday at Defcon, so this is good timing.

I agree with many of Adrian’s criticisms, including that Visa could have just given more specific guidance overall.  But I also understand Visa’s need to keep the guidance vague enough so as not to provide undue direction to what is basically a fledgling market space.   Which is exactly where I see the tie in with Josh Corman’s primary argument about the PCI Council; intentionally or not, they are steering the security market space through the PCI standards.  Visa could be a force for good in the tokenization and truncation markets if they predict correctly and back solutions that are for the best over the long term.  Or they could be seen as stifling innovation if they issue poor guidance.  Much like the PCI Council.

Earlier today I heard someone make the statement that the majority of companies who are compromised are using encryption in some form, but they still got compromised.  He was reminding me that none of the other silver bullet’s we’ve thought would save us from the bad guys have worked, so use truncation and tokenization, but know they won’t solve all our security issues.  As is so often the case, they’ll just move the attack to other targets and use other vectors.

If you’ve been thinking about using tokenization or truncation to limit the scope of your PCI environment, you need take a few minutes to read the two documents Visa just released, Visa Best Practices: Tokenization and Visa Best Practices for Primary Account Number Storage and Truncation.  Neither of these documents are more than four pages in length, so they only take a few minutes to read, but they give you a good starting place for asking questions about both of these market spaces.  There’s nothing exciting or unexpected in either of these documents and you’ll need to do a lot more research to understand the more complex elements of both solutions, especially as they relate to your specific environment. 

If you’re part of a merchant organization or somehow dealing with credit card numbers and you’re not considering tokenization or truncation, why not?  Is it lack of time, lack of resources, lack of management backing or something else?  Have these technologies simply not risen to the level where you felt the need to take them seriously?  I’m curious as to why you might not be looking at a technology that could limit the amount of sensitive information on your network.  I’ve talked to a number of merchants over the last year and there’s been plenty of interest in the ideas of tokenization and truncation, but I’ve only seen a few merchants actually making a move towards implementation.

I hope the next guidance we’ll see comes from the PCI Council, giving instructions on how both of these technologies can be used to reduce the scope of a PCI assessment.  What can you take out of scope?  What common mistakes might bring systems back into scope?  What should we be looking for in an implementation?  These are still relatively new technologies, the implementations differ significantly enough that greater direction and care are going to be needed in their assessment and validation.  There are some things that are laid out in the Visa documents, but I think we need to look for more specific guidance from the Council.

Rich and Zach are still sweltering in their perspective heat waves, but Martin managed to nab an interview with Bob Russo, the head of the PCI Security Standards Council. We also cover a couple of stories and some honest to goodness listener mail!

Network Security Podcast, Episode 205, July 13, 2010
Time:  44:44

Show Notes:

• Listener Mail
• Interview with Bob Russo
• FBI Raids ‘Electronik Tribulation Army’ Over Witness Intimidation.
• Letter to the client.
• Tonight’s Music:  Missing You by Blue Matters

Last week another assessor friend of mine started a new blog, Fear Not the Assessor.  She started it off with an excellent post, Letter to the Client.  Almost every QSA goes into a new client with a certain sense of trepidation due to client’s preconceived notions and most merchants going into an assessment for the first time are nervous because they don’t know what to expect, all they know is what they’ve read online.  That first phone call with the client is always so much fun for everyone involved.  The Letter attacks some of those notions and list some of the steps a client should be taking before the QSA ever comes on site.  As a way of introduction, a letter like this really helps put many clients at ease, letting them know that you’re there to help and not simply pass judgment on them. 

Here’s a letter of my own with several more points to ponder.

Dear Client,

We’re about to start on an effort of many months of work that both of us hope will culminate in the issuance of a compliant Report on Compliance.  There will be surprises and setbacks along the way, but I’m sure that we can work together to overcome them.  My job is to help assess the security of your cardholder environment and provide you with honest feedback about your compliance with the PCI standards.  Your job is to provide me with the information I need to make that assessment.  Together we will document your environment and show that it is both secure and compliant.

Several things you should know:

1. Securing your data and your network should be the goal and PCI is just a signpost along the way.  Please, please, please don’t make the mistake of thinking once you pass your assessment that you’re secure and you have no more work to do until next year.  PCI is a good starting point for securing your environment, but each company is so unique that there are innumerable holes it leaves open to exploitation.  And the assessment only covers your cardholder data environment: what about the rest of your network?
2. I am judge, but I am not jury nor executioner.  I will make judgment calls on the state of your environment and I may find things I do not believe are compliant.  You may agree or you may think your controls and safeguards are sufficient.  Make your case to me, and if we still don’t agree, we can bring in other QSA’s within my company to review the situation, starting with my manager.  Sometimes they’ll see something I didn’t. 
3. I will never leave you wondering if I found something wrong.   I will always try to let you know at the end of the day, if not at the end of each meeting, if I have any questions or concerns.  It’s in both of our best interests for me to be as transparent as possible.  The sooner you know of an issue, the sooner you can begin investigating and getting it resolved.
4. You are my client and it is my job to help you receive a compliant RoC.  I will give you the best advice I can to help you achieve compliance.  But it is up to you to establish the policies, procedures and controls needed to reach this goal.  If I identify a requirement that is not being met, I will bring it to your attention and help you address the issue in a timely and cost conscious manner.

Clear communication is a good salve for many of the pains an annual PCI assessment brings.  I look forward to learning about your company, your network and your people.  And I hope that the lessons I’ve learned helping dozens of companies become compliant can be used to help you avoid some of the pitfalls and false starts of compliance.

Truth can be stranger than fiction sometimes; I’ll be speaking on a panel on compliance with Jack Daniels and Josh Corman at Defcon next month.  There’s a couple other people on the panel, who I’ll add once they’ve been confirmed.  This should be a fun panel, since we won’t be as interested in keeping it completely civil as we would at someplace like RSA or BSides.  We’ll laugh and shake hands afterward, but don’t be surprised by anything you hear during the panel.  And this is an interesting crowd to give this talk to, much more technical and focused than more managerial conventions like Black Hat.

I talk to Jack, Josh and a lot of other people about PCI fairly regularly.  I’m fairly confident I know their positions on compliance and they have a good idea of mine as well.  Jack’s a good moderate who sees both the good and bad, while Josh sees it as a tidal force in the security market space, and not one he likes.  Where PCI points, the money goes, like it or not.  But this talk won’t just be about PCI, we’ll talk about compliance in general, the good, the bad and the ugly. 

If you, by some chance, are around at Noon on Sunday, come see the discussion.  The question I have for the audience is simple, “How has compliance affected you and/or your company?”  Has it’s affect been positive or negative? Given the crowd we’re drawing our audience from, it could generate some very interesting responses.  I’m curious to see how a group that collectively thinks of themselves as hackers feels business attempts at compliance frameworks really affect the work they do.  I expect to hear more annoyance with compliance getting in the way of real work than anything else.

This should be a fun way to end Black Hat and Defcon.  Josh and I really haven’t had it out over whether compliance being a market force is a good thing or a bad thing and this is a good venue to draw him out on the subject.  I’m looking forward to it.

It can be downright disheartening to be a QSA.  If you do your job and identify holes in a merchant or service provider’s systems, they’re upset.  If you try to help them adapt their current systems to meet with PCI, they think you’re letting them off the hook.  If you send them a packet of documents about what to expect during the assessment and what they’ll need to gather, more often than not the client will ignore it and claim you never told them what you needed.  If their due date for compliance is coming up quick, it won’t matter how long you told them the writing and quality control process would take, they want their Report on Compliance turned around overnight.  And then there’s the whole ‘check list’ mentality that has many people responding to the letter of the PCI DSS, completely ignoring that with a little more effort they could have increased their security instead of just marking off a box.  Yes, being a PCI can be frustrating, annoying as hell and will burn you out if you’re not careful.  Just ask my friend Michelle, she’ll tell you exactly how hard it is to be a Qualified Security Assessor.

She’s got a number of good points; we see all too many clients who just want to have their PCI assessment and then ignore the whole thing for the next 8-10 months, until the whole process starts over again.  They don’t want to think about PCI at all during that time, they don’t realize that there are a number of requirements that mandate continuing effort on a daily basis, not just when the assessor is on site.  And we never, ever see clients putting lipstick on the pig just to cover up a deficiency until the assessor is gone.  Oh no, never that.

But there is an upside to being a QSA.  Some security departments have learned to do an awful lot with very limited budgets.  Some clients understand that attaining compliance as a side effect of security is actually cheaper and easier than trying to do it the other way around.  Some clients actually want an honest review of their environment that identifies potential weaknesses outside of a strict interpretation of PCI.  And every so often you run into a client who’s doing something unique and unusual that doesn’t meet the letter of the law of PCI but still manages to exceed the intent of the requirements, sometimes by quite a bit.

These are the clients who keep me from pulling my hair out.  I find it rejuvenating to talk to a client about the security impacts of changes to their environment honestly, rather than trying to argue an interpretation of PCI that doesn’t require them to make any changes or worse, leaves them less secure if implemented.  When a client understands their own environment and knows why their data is where it is, it makes my job, and theirs, so much easier than when clients are doing their discovery while I’m on-site.  And sometimes I’m actually working with a client to secure their environment, rather than fighting to get them to implement basic security controls.

I recognize that being a PCI QSA and consulting with clients on meeting the DSS requirements is a balancing act; we try to balance security against the DSS against budgetary and manpower constraints.  And since we only have two hands, balancing three competing limitations is hard, very hard.  If you’re in this field and you don’t feel burnt out from time to time, it means you don’t care.  And that is probably a bigger vulnerability than most of the technical requirements in any compliance framework.

It’s the clients who view the security of their company as a calling that keep me coming back.  It’s easy to check off a box, go home at night and ignore what’s happening to your business while you’re away.  But some security professionals are intensely passionate about what they do and how well they’re protecting their enterprise.  These are the people who make being an assessor worthwhile.  Because even if you’re arguing with them about an interpretation or commiserating about a requirement that sounds stupid on the surface, you know these people care and at the end of the day, they don’t just walk away thinking their job is done, they worry about bettering their company’s security the next day.

Next post I’ll address Branden William’s post “Why ISA’s are qood for QSA’s“  Can you say “arm chair quarterbacks”?  I knew you could.

I’m a big fan of tokenization and end to end encryption (E2E2).  Never mind the fact that neither technology is fully developed, nor do we even have a real definition of either technology.  The fact that both of these technologies have the potential to take credit card information our of the general merchant environment and gives the bad guys less reason to attack is enough for me.  It won’t stop attacks against merchants all together, but it will cut down on the value of breaking in and therefore cut down on the number of attacks, at least in theory.  It will also cut down on merchants’ responsibility for meeting with the PCI DSS requirements, since much of environment that the QSA’s have to review will now be out of scope.  But without the threat of PCI (and potential fines/fee increases) will merchants keep up the minimum security safeguards that PCI mandated or will they revert to their old ways and ignore security for the most part?

One of the big questions that comes up over and over again is how effective is PCI in securing the merchant environment.  And the answer is, no one really knows.  Breach disclosure laws prior to 2003 were non-existent, and even once California passed SB1386 and got the legal ball rolling, breach disclosures have been spotty at best.  Now that we’ve got some 40 states that have some form of breach disclosure law, the information we’re able to gather is much more consistent.  Unluckily, we still lost the ability to have any real baseline to measure the success of PCI against and anyone who says that PCI is or isn’t effective is mostly going on their own anecdotal evidence, not hard data.  Verizon’s Incident Metrics Framework may help in gathering statistics going forward, but we’ve already lost the data needed to measure the effectiveness of PCI.  (Disclaimer:  I work as a QSA for Verizon Business)

As tokenization and E2E2 take hold, we’re going to have another chance to see how effective PCI is in securing the merchant environment and whether or not merchants are really going to secure their environment without the threat of PCI hanging over their heads.  There’s almost nothing in PCI that a shop with a good security program shouldn’t be doing in the first place.  Firewall reviews, anti-virus, log monitoring, IDS, etc. are all safeguards that are mandated by PCI but are security measures that any good security shop should be putting in place for their organization by default.  The fact that many organizations couldn’t get the funding for some of these tools until PCI came along is a measure of how hard it is to get the budget for security.  And if organizations start losing the funding for these projects because tokenization and E2E2 have taken the majority of their systems out of the scope of PCI, we’ll know that PCI was the real driver for the safeguards, not any real concerns over security.

PCI is expensive.  Security is expensive.  Not necessarily because the tools are expensive, but because merchants ignored security for years and have had to spend a lot of money and time to implement the tools they should have been running in the first place.  If they can reduce the scope of the systems they have to protect through new technologies and no longer have to be assessed on an annual basis, do you think they’re going to keep paying for the tools that they implemented just for compliance or do you think they’re going to let their IDS and log management tools fall by the wayside?  I know that some of the shops I’ve seen will keep the tools and keep using them properly.  But I think the majority of merchants are going to go back to their old ways and do the bare minimum that their security group can fight to keep.  If your company’s marketing department depends on PCI to make sales, I’d be very afraid of tokenization and end-to-end encryption.

On Twitter this morning, @secrunner made the following comment:

“I think it’s surprising that PCI still hasn’t developed a program to certify pen testers or at least standardize the approach”

In reply I stated that given the level of certification for ASV’s (Approved Scanning Vendors), I’m just as happy if the PCI Council would stay out of the business of certifying pen testers or creating a standardized approach.  In reply @secrunner asked the following:

“In the spirit of PCI, isn’t even some standard (even a low one) better than none?”

The answer is, no, it’s not.  To be more specific, my answer was “Low standards for a merchant are better than nothing.  Low standards for a vendor are misleading at best, dangerous at worst.”  Let me explain why I think this way:

When you go shopping, one of the last things on your mind is probably “How does this merchant protect my cardholder information?”.  It’s one of the first things I think of, but that’s what I do for a living.  Most people are just concerned about if their merchant is going to have their size or the best price on the new toy they want.  They just assume the merchant has taken the necessary steps to secure their cardholder information.  And if they haven’t, consumers know that they’re only responsible for the first $50 dollars worth of fraud, and even that is usually absorbed by their bank or credit card company.  Sure, getting a new card issued to you is a bit of a hassle, but for most people it’s something that’s over and done with in a few minutes.

In this case, security is assumed and is not the primary concern of the person doing the purchasing.  A default standard such as the Payment Card Industry (PCI) Data Security Standards (DSS) is important and useful because it gives a baseline level of security for the industry to meet.  It may not be the level of security the company really needs to protect themselves, but all too often this baseline is more than the company was doing prior to the standard.  It may not be perfect security, but at least it pulls you up from the level of ‘low hanging fruit’. 

Certifying a vendor as a ‘compliant’ or ‘certified’ is a completely different story.  When an industry group such as the PCI Council makes a standard for a class of vendor and then certifies these vendors as meeting a certain baseline, this certification becomes one of the primary influencers in the purchasing decision.  Using the ASV certification as an example, a merchant won’t even consider a scanning vendor for their company unless the PCI Council has already certified them.  The merchant has to use a vendor who’s been certified otherwise they can’t submit the scans as part of their own compliance.  A large part of why this works is that external scanning of web sites is a fairly well understood, repeatable and, most importantly, testable process.  It can be easily automated and running the same test against the same site ten times will generally generate the same results every time (okay, maybe 90% of the time)

Penetration testing is an entirely different issue.  Yes, there are automated tools.  Yes, some pen testers don’t go much beyond that level.  But the good pen testers I know treat penetrating a company’s defenses more like an art than a science.  Metasploit and other tools are their paintbrushes, but it’s the person who’s using the tools that is actually making it possible to find the vulnerabilities in your company so that you can shore up your weaknesses and prevent someone else from finding them.  This isn’t a process that easily documented, standardized or testable.  It might be something you can do on a person by person basis, just as the PCI Council does for QSA’s now, but it would be nearly impossible to do for a company.

Let’s be honest, in the PCI-DSS, the idea of ‘penetration test’ is barely even defined.  It has to have a network portion and an application portion, but the how’s and what’s of penetration testing are left up to the QSA to verify and validate.  There’s no agreed upon standard in the industry of what makes a pen test a valid and acceptable pen test, let alone within the PCI community.  If the PCI Council wanted to certify pen testing companies, the first major hurdle they’d run into is making up that definition.  Then they’d have to come up with a way of testing companies’ adherence to the standards and create a certification program.  This would be a huge battle to undertake and the benefits would be minimal. 

Right now, it’s up to market pressures and QSA’s to determine what’s a ‘real’ penetration test.  If someone created a penetration testing certification there’s only one group of people it’d help:  marketing.  Most merchants wouldn’t read the requirements for the certification, they’d just use the certification process as a check box to weed out potential vendors.  And I can guarantee that the marketing teams would love that.  And I doubt it would make the results of penetration tests any better; in my opinion it would simply mean that most companies would ‘dumb down’ whatever they’re currently doing so that it met with the minimum standards and no more.  I much prefer seeing the merchant who’s having the pen test performed ask questions about exactly what’s going to be done and try to understand what they’re getting themselves in for.

Do you remember those old School House Rock commercials from the 70′s?  I do, in part because someone gave my kids a DVD set with all of them on it.  And apparently the folks at the PCI Council remember them too, because they’ve created a video that looks a lot like those old commercials.  My favorite part is the fact that Bob Russo let a cartoon version of himself be part of the video.  I wonder if the real Mr. Russo can sing and play the guitar? 

Unluckily, the only time I was able to make it down to SF Bsides was for the Great PCI Debate, part 2.  Luckily, all the rest of the presentations that went on there are available via Ustream.  Of course, I still say the Great PCI debate was the most important presentation, partly because it contains guest spot by me (and several examples of me yelling from the sidelines).  There was a momentary glitch where the video stream was lost for a minute or two, which is why it’s in two separate parts.  In any case, watch my friends, Jack Daniel, Josh Corman, Andy Ellis, Michele Klinger and Anton Chuvakin discuss compliance in general, not just PCI.