Now, some direct answers to your questions:

1) No, the target apps were not already tested with a smart fuzzer. We chose a binary parser and a text parser which hadn’t been fuzzed before, so we weren’t retreading old ground. We’ve also done additional smart vs. dumb testing on some older parsers, and the results were similar. We’ve tested a number of internal AND external tools to try to normalize across fuzzers that may not be very good.
2) We tested file parsers, and granted, state doesn’t matter quite as much as it would for protocols. It still takes a lot of man power to build format specs for a network protocol. In addition to smart fuzzing, there are a number or MiTM fuzzers that allow for random manipulation in-line between host and client, after a simple capture/proxy setup.

As I understand your tool, you’ve already done the work for a customer in making it intelligent, and so their investment is x numbers of dollars, and some training, not building a format spec from scratch (human hours). You’ve essentially amortized the cost of building the formats across a large customer base, and your software finds bugs. For many folks, this makes good sense.

The presentation will be available for public viewing on technet before too long.

p.s. Ari, if you want to chat, ask Juha, Josh, or Heikki for my contact info.

I know it’s not exactly intuitive, but the idea is that the effort involved in writing a grammar for whatever is being fuzzed (which it’s always possible – it could be intractable), and the time it takes to execute said “intelligent” tests, just doesn’t match up to the time taken to do random testing.

One of the things the MSFT guys showed at the con was bugs discovered were a factor of test cases (pretty obvious here), not on what test cases were selected (the “random” part). The only caveat(s) were the test template selection (be sure to select files that dont overlap testcases) as you want high code-coverage, but you *do* want to be fuzzing between the gaps, which intelligent fuzzing misses.

If/when the presentation gets posted I’ll link to it here as it’s certainly worthwhile watching. In the mean-time, this presentation probably give a little more info on your position. http://video.google.com/videoplay?docid=-6109656047520640962&hl=en

I can’t believe MSFT (and you) think that could be true! I think this has been proven to be the opposite a number of times. The goodness of some “intelligent” fuzzers can definitely be argued, but overall a truly intelligent fuzzer will definitely be much more effective (in both time and number) in finding flaws. I would really like to see the data behind this claim.

There are two cases where this could be true:

1) Someone is already using an intelligent fuzzer. Re-testing it with another one (a worse one) will not bring much benefit. A random fuzzer can still find a flaw (with a lot of luck and time). A better intelligent fuzzer will of course also find flaws, and more than random fuzzer, again.

2) The “protocol” you are testing is stateless and flat. A random fuzzer can easily test this (and only this one could argue), whereas an intelligent fuzzer will not find much challenge there. This is very rare case though. Try fuzzing stateful protocols like TCP or SSL/TLS, or maybe even some ASN.1 telco protocols; and a random fuzzer will prove to be completely useless in comparison to an intelligent protocol-aware fuzzer.