Seven Deadly Pen Test Sins

March 15, 2008

Seems that this post has made it’s way to a number of places before I’ve got to post it (sorry – been busy ).  However, it’s bang on target, and I though I’d add my own observations as well.

Managing Time

One of the most common yarns about the difference between pen testing and hackers is that pen testers have a limited amount of time to look for vulnerabilities in a small window of time in the applications’ life. As a result, time is precious. Knowing when you are on to something that you can confirm in a reasonable period of time is probably the biggest place where good pen-testers go bad. It is really easy to turn a pen-test into a research project. And while you toil away on a tiny sub-section of the application, you may never get to that remote code execution flaw that is lurking elsewhere. This is also where coverage vs. depth plays a huge role.

This can be really common, and is why I’m a big advocate of using a methodology/checklist.  Not only does it ensure that a base-level of testing is being performed, but it also allows you to track what has been done vs. what hasn’t and ensure that you focus on getting everything looked at within the time allowed.

Smugness

Devaluing findings that customers care about. Yes, XSS and CSRF are lame findings to people used to exploiting memory corruption or even compared to SQL injection and AUTH bypass. This also extends to “I found one, let the dev team find the rest of them”. Smugness can also be extended into overconfidence. And overconfidence equals underestimation. This all results into missing vulnerabilities that you needed to find.

Sometimes the inverse is true, especially when there is a client that really doesn’t have much of a clue or isn’t all that interested in security.  Fortunately, when client approach us at Foundstone, the vast majority of the time they “care” about the results, but unfortunately in some instances it’s just to “pass some process” and that can (in my experience) be especially true with PCI-type engagements.  Just because I am fluent in all the potential attack vectors and vulnerabilities for a given platform, doesnt mean the client it.  It’s therefore really important to put yourself in the clients shoes as often as you can.

When it comes with the “I found one [vuln], let the dev team find the rest”, at one level I agree with the sentiment, but when combined to the managing time point above, we can have a disconnect – if the application has systemic problems, chasing each and every one down also means that not all of the application gets tested.  In that case it’s best to clarify what the approach should be.

Never understanding the app

It is easy to just treat an application as a series of inputs, and not bother to understand what the application is actually for or what it is actually doing under the hood. Good penetration testers are often trying to get into the developers’ heads.

Getting into the developers heads is, IMHO the only way that you will ever find the really important findings.  It’s really easy to find the usual “low hanging fruit” or “best practices”, but the real high risk stuff is where business logic fails and allow an attacker access to data they should have, or subverts flow to do something they shouldn’t like obtain something for less value than it should be, or other common issues.

Over-automation

While I am a big fan of utilizing tools to make people more effective, there are two problems with relying on these tools:
1. They generally create as much work as they eliminate. False positives in the popular web app scanning tools are still common enough that you waste a lot of time, especially on a small website.
2. After you run them, you don’t have any better understanding about the application. It encourages #3.

One of the big, sometimes unwritten, problems with webapp scanners is their huge false positive rates.  Often I run one of the well-known scanners only to have 100+ “findings” of discovered directories, login page discovered, and various other useless information findings, but rarely anything that I’ve not already found manually while waiting for the tool to complete (usually for an “average” size site 3-4 hours of scanning isn’t uncommon).  As I said in Mark Curphey’s blog way back, “tools do not make the man” – any pen tester should know the tools, their pros and cons, why to use one and why not, as well as not only how the tool works, but how they would dev the tool if they had to – hire the people that write the tools rather than run them!

Sloth

This raises its ugly head in a couple of ways. It is usually either in avoiding the difficult parts of the test or conversely the easy parts of the test. Total human nature. One is hard and the other is boring. As a result, usually the team that comes in after this sinner finds serious flaws that are either hard-core or are embarrassing.

Sometimes this one surprises me.  Based on the number of apps I’ve tested over the years, I’m sure that there are been some things that I’ve missed but hopefully none of them serious.  Having a checklist, understanding the app, and watching the available time (see above for all of these) helps ensure that you cover all the bases.  If in doubt, it’s always worth going back and asking for more time – either increasing the scope of the work, or getting “internal” help to see if it’s worth following your nose on something.

Stagnation

Given the difficulties of the job, it is easy to not evolve one’s skill set. This is compounded by the fact that even in 2008 there still aren’t enough resources out there to keep evolving. This is also an organizational problem inside of every company. It is why the evil M word raises its ugly head.

If there’s one that that’s a “con” in the work of a security consultant is that it’s very, very easy to get “stuck” doing the same thing over and over, never really “digging in” to anything.  Mark once again has a good post on it here.  In some ways the security industry moves quite fast, but in any “focused” area (like webappsec) unless there’s a big new bit of research or “idea”, the same old findings are still there – I’m still finding the same SQLi and XSS as I was 4-5 years ago.

Communication/Soft Skills

Where it’s a project manager or the customer, you need to understand what the customer cares about, manage their expectations, and oh yeah — if you can’t actually care, at least pretend to. Lots of people think that doing a good job is simply breaking the app. That is enough until someone who is as technically savvy as you rolls in and also has that polish. (You know, the one that you can’t stand).

I have to remember this each time I do an engagement, and it really holds me in good sted – clients aren’t hiring you/your company’s services for how much a 1337 h4×0r you are – it’s the work and the experience they get out of it.  Remember, generally the only thing, other than your time, that a client get’s out of an engagement is a report – tens of thousands of $$$’s sometime for a nice stack of (not even printed all the time) pages.  The cost per page therefore is pretty expensive if you just think they are paying for pages upon pages of findings/vulnerabilities.  The “value add” that they are paying for is not only the skill of the people doing the testing (that’s “assumed” after the sales guy has got them to sign on the dotted line), but understanding their business, their problems, and their solution.  Business + Technical excellence is how our VP kept drumming it in. 

In summary, this is an excellent list of things to remember for anyone, whether pen testing or QA testing.  Keeping all these things in mind only make projects run better and ensure the client gets the most out of the engagements

Posted in Security