Does SDL work?

April 24, 2008

There’s been some good discussion questioning the merits (and metrics) of the Microsoft SDL over the past few weeks, and I’m glad that I held off on writing about it here as there’s been other people giving their views.

Pete Lindstrom kicked it all off with his post “Microsoft’s SDL has Saved the World!!“.  The basic argument here was that looking at the number of vulnerabilities disclosed after a product has shipped is not a good way of measuring a product’s security, or if the method used in improving that security is working.  This is a difficult line to tread, as one of the problems with software security is that there are no good metrics to use to evaluate “goodness” other than bug counts.  Although bug counts are used internally during development to track the progress, quality, and robustness of software, it’s only when it gets widely deployed that anyone tracks vulnerabilities.  Before that “release”, software is often in flux and lots of changes are being made (some last minute, some that require the scheduled release to be “pushed” because it’s not of the required standard – MSFT has done this a few times) which may improve, or sometimes decrease, the overall security.  From a software companies point of view you are trying to protect your customers (and not the developers of the product); there’s not IMO any other time that you really can start tracking other than from the release date.

However, Pete says (and responds) that because Microsoft are basically employing all the people that would review a products security after release, and bringing them into the development process (the SDL – other companies are brought in as part of an external review), the vulnerability count is artificially low.  The challenge is that because Microsoft has hired everyone that will look at their products critically, there’s no-one left to log any vulnerabilities after release.  It’s a good argument, and I certainly can agree with his points, but it was fervently denied by one of the best vulnerability hunters out there, David Litchfield, as well as the most public face of Microsoft’s SDL, Michael Howard.

So, does the SDL (Microsoft’s or any other) work?  Pete himself says that he believes it does work, but the wrong metrics are being used to show this.

I think the thing to keep in focus is what any SDL tries to do – build security into the development at all stages.  This certainly is what Microsoft’s SDL does, and is the most comprehensive I’ve seen.  We don’t have many good security metrics, from software quality through to ROI, so that is a worthy area of investigation for people smarter than I am.  What is clear to me though is that some companies (and Microsoft specifically) are taking security seriously and putting processes in to improve it, whereas a lot of others are just ignoring the problem or are doing very little to address it.  How to measure any improvement (or lack of improvement) is rightly up for debate, but it certainly “feels” that security is getting better in MSFT’s products and doing something is clearly better than doing nothing.  As Bruce Schneier says, sometimes just the feeling of security and the actual security match, and sometimes they don’t but it still works.

Posted in Security