Bug reports and methodologies

November 19, 2008

I’m not sure where this link resurfaced from – I saved it to read and got to it the other day – but this post from Joel on Software has two of the things I spend many a day looking at – bug reports and methodologies.

Bug reporting

Everyone knows how to report a bug right?  Repro steps, symptoms, etc, etc.  Just as everyone knows how to fix them – reproduce, find the root cause, fix it, test the fix.  However, there’s two parts on both sides of the equation (test/dev) that I’ve often seen left out.  Testers don’t show enough evidence so devs know it’s and issue to fix (so mark as non-repro), and developers don’t go back into the rest of the code to find/fix similar problems (they just fix the issue they have been provided).

This happens a few time a year with me – I’ve found some vuln (say XSS) that is systemic in the application, but it’s pushed back as not an issue or non-repro, or if it does get fixed it’s only with the specific test case I supplied.  So, here’s my recommendations in dealing with that.

For every (well, non trivial at least) high-risk vulnerability I discover when doing a pen test, I do a screen recording of it.  That video helps in a number of ways.  Firstly, it specifically details the reproduction steps in a way that can’t be misconstrued (and I do a voice-over explaining why I’m doing it), and also it show the issue in a way that it can’t be dismissed as non-repro – the dev/PM can see it happening on at least my machine, so it’s valid in at least one situation/environment and worthy of investigation if it’s not the same in the developers.

I’ve used Techsmith’s Camtasia product for screen captures from way back when I had a copy at FloridaTech.  It’s not cheap, and I should really upgrade it now, but even the older version works well enough for me for what I need it to do.  If cost is an issue to you, try Microsoft’s Windows Media Encoder – it’s not as simple, but will still get the job done.

Make the decision yourself – would you rather watch a vuln or read a description like the following

On the "buy a book" page, the quantity drop down isn’t validated and can be set to a negative integer.  Doing so, an attacker is able to purchase a negative quantity of books, and therefore obtain a refund on their credit card.

What is possible with this description is that it’s lacking reproduction steps – the developer could look at the dropdown and just say that a negative number can’t be chosen.  Bug closed – no repro.

Although generally it’s not the testers role, but when Foundstone logs a finding we always provide a detailed recommendation for each issue discovered.  This gives the developer an idea of how to fix it in a way that isn’t just to patch that one specific occurrence.  That way you encourage "correct" behavior (i.e. centralized validation, consistent access checks, etc), and for the developer to go off looking elsewhere for similar issues.  Doing this for black-box testing is sometimes difficult (you lack the context), but it at least provides the developer somewhere to start.  In saying this however, developers clearly have more insight into the inner workings, know the app better, and might be really experienced in what to do, so the recommendation provided may not be acted upon in the exact way, or perhaps ignored completly for another approach (including doing nothing and just accepting the risk) which leads me nicely on to methodologies.

Methodologies are not for idiots

One of the things in Joel’s post that I didn’t agree with is that methodologies turn people into compliance monkeys.  At Foundstone we have methodologies for most (not all) of the service lines we deliver, and there’s a reason for that – without a methodology to "guide" a tester (even an experienced one), it’s very easy to go "off down rabbit holes" chasing a possibly promising bug, and end up with little time to test all the things we are supposed to look at on an engagement.  Methodologies keep people on track (even if it is looking at "best practices" a post on that little topic soon!), provide a means for a baseline of activities, and (as Joel points out) can be used as to ensure a repeatable process.

However, in saying that, there are some issues with methodologies that are pointed out.  For experienced testers, the last thing they want to do it get caught in the weeds – detailed descriptions of what to look for/do are fine for inexperienced tester where the guidance helps them, but just get in the way of experts, who really just need a checklist of things to do without the "fluff".  Also, there has to be room (and encouragement) to go off the methodology – no test plan is totally complete/sufficient, so ad-hoc testing in areas that perhaps aren’t in a methodology in important as well.  When I’m writing a methodology at Foundstone, my "test cases" (for want of a better term) look like this.

Perform TCP sweep on the target server.  Note any extraneous open ports.  Optionally add findings for any open ports other than 80 and 443 if external test.  Be aware of duplicate findings if importing Foundstone Enterprise scan

****************************************************************
Why:
Web servers should only communicate on two ports – 80 and 443.  Any additional ports open to the outside world expose services that may reveal a greater attack surface of the server/application than is necessary (much of the "internal" services a web server may require for maintenance/operation such as SSH, SMTP, etc should be firewalled off).  Most modern web servers also have administrative ports that should not be available remotly, but sometimes due to misconfiguration are – allowing access to these, even if authentication is required may be considerable risk to the server.

How:
Run a full TCP port scan with Foundstone Enterprise/NMAP.  Note: this is often not the default behavior for any of these tools. check the settings of the tool you are using. For TCP source port scan, ports 20, 53, 67, 80, and 88 are most likely to give fruitful results.

1. Foundstone Enterprise set the TCP Scanning dropdown in the services settings to "all"
2. nMap use -sT -p1-65535 (or -sS -p- if you just want a "quick" scan, but be aware of the limitations)
3. Under some circumstances, changing the source port will allow for connection through a firewall.  Use the -g flag in nMap to do this.

For more info on nMap options, see http://nmap.org/man/

The sections increase in detail so that the tester can get the level of info they need.  An experienced tester just reads the introduction so they can remember what this test is for and they know to stop reading when they see the asterisks.  Perhaps I need refreshing on why I’m doing this test or what to look for – for this I look at the "why" section.  Finally, if I’m a novice tester and don’t know how to perform the test case the "how" section details it in steps (including tool commands/parameters if appropriate, although these are auto-configured in the tool I’ve written to help guide testing so the user should just be able to hit "play").  Different sections of the methodology use information gathered from previous tests, or can be used to "confirm" valid results and test completeness.

In any case, I’m glad that I came across that old post from Joel, and really like reading his insightful material both online, in Inc magazine, and the stuff on StackOverflow, so please add them to your RSS feeds.

Posted in Musings, Security