The futility of black-box testing (in some instances)?
November 25, 2008
Consulting can be a lonely job at times – often we are either on a client site, or working at home (which don’t get me wrong, has it’s own benefits) – so having a chat open between all the other people in Foundstone keeps one "connected". Although the beer-signal-to-noise-noise ratio is sometimes low, it’s really good being able to tap into the "hive mind" occasionally.
On the chat today, one of my colleagues is testing a (traditional client-server) app and it turns out that many of the inputs/output are encoded. We share our favorite encoding/decoding tools, but a through springs to my mind – what if everything, absolutely everything, inputs, outputs, cookies, etc, etc, were encoded, in an app and not just in some straightforward way like base64. Perhaps in the programmer’s own "custom" encoding. From black-box testing, how easily could this be tested?
Tools would simply fail; unless you owned the tool, or knew it well (enough to tweak it, what can/can’t work), their "dictionary" of tests just wouldn’t work – each "attack" would have to be encoded/decoded either in place or on the fly to be understood by the server (/app) or client (/browser).
Manual testers would often just give up – the overhead of figuring everything out, crafting various attacks, running them, modifying them, etc, etc, could just get too large to manage. Most testing (either pen tests or QA) is time-boxed anyway, so would this "encode everything" route simply run out the clock. Things that a tester might get round to looking at, or identify as needed to be tested could just get missed. (I appreciate that this is just as importantly a scoping issue for you contract/billable-hours people out there, but just run with this for now ok 
)
Would attackers have the same problems? Unless it’s a big, juicy target (like Google/Microsoft for example), would the investment pay off for the pain? Would they also give up and look for easier targets? Does, in this case, security by obscurity pay off to some degree? Remember we are not talking encryption here, encoding only.
Now, let’s consider the same app from a white-box approach. I posit that it’s *much* easier to test such an app. Once data (inputs/outputs) are inside the code, it’s easier to track how they are being used. Even if they are kept in their encoded state until the very last minute (or not even decoded at all), it’s much easier to spot code constructs that are "bad". Think, as a simple example, SQL injection – seeing a query being created on-the-fly may not actually be SQL inject-able, but it certainly is a cue to look into it a bit deeper. Same with many of the other vulnerabilities.
Perhaps I’m even evil enough to modify one of the hacme apps, or some of the other code I have lying around, to be such an "encode everything", and submit it to some people and tools for testing. Anyone care to guess what the outcome would be? Perhaps even engage in this little experiment themselves?
I don’t believe that black-box testing will go away – it’s too engrained into our industry/culture, and it gives a good bang-for-the-buck if used correctly. However, I think that this is at least a good argument for grey-box testing. Having the code available to a tester, even in a non-compileable state, just to look at when necessary, would help immensely in these "what the f is this this app doing" situations, and quickly identify some problem areas to look are more deeply rather than going down the preverbal rabbit-hole.
Posted in 
