Automated security testing and its limitations
November 24, 2008
A nice post over at ITPro about automated (web) security testing.
Nothing new for the people that follow this field, but interesting that the author sees about a 25-30% positive finding rate, and clearly identifies some of the things that the tools miss.
One of the things he mentions just makes me sad though
This is quite annoying in the context that some (thankfully not all) independant commercial testers we have encountered over the years believe an automated test is a sufficient test to charge a fortune for, and rely wholly on. If we had done with with our internal tests we would have missed several critical “information disclosure” type bugs which were remarkably simple to spot.
These “scan boys” as Curphey used to call them give independent testers/consultants a bad name. Sure, tools are an important part of any testers arsenal, and are required as a “belt and braces” approach (how stupid would you look if you missed an issue that an automated tool discovered), but simply running a tool and thinking that is sufficient? What is the client paying for that they wouldn’t get by just running the tool themselves?
Foundstone uses various automated (both web and network) tools at the beginning of a test, mostly for information gathering, discovery, and finding the extremely low-hanging fruit, the rest is performed manually by a well trained, knowledgeable, consultant. However, even if a tool says it tests for something, part of our methodology is to go back and verify that by hand. Tools can be wrong after all.
Automated testing tools, especially in the web world, are not at the level where they can be used to find even most of the issues in an application. Myself and others don’t think it will ever get that far. For that reason, I don’t think we can “fix” security by “testing it in at the end” – we must build software with the characteristics we want from the beginning, and not patched in from (black-box) test findings.
Posted in 
