What I’ve found interesting in this case is who benefits from click fraud.  In the case of Google’s AdSense or the Yahoo publisher network, or the many different others, when you share the revenue of click-through’s of ads with the people that host them, there’s an incentive to drive clicks either via legitimate (increasing traffic), semi-legitimate (SEO techniques), or illegitimate (click-fraud) means.  More people going though the ads you get served up, the more money you make.  What is interesting on Facebook’s current issues is that they are hosting on their own site, and aren’t sharing revenue (well, they are with application developers, but for brevity I’ll ignore that instance as I believe it’s manageable).  So in that case, who’s doing the click-fraud and for what purpose? 

On TechCrunch there’s an excellent post on how click-fraud may be working against Facebook.  In a nutshell, advertisers are probably click-frauding each other to drive their competitors out of the market by using up their budget and/or spiking the prices. If this is actually what is happening (I’ve seen no hard evidence out there one-way-or-the-other yet from crawling around the web doing some research before putting this post up, but stealing a link from one of the TC comments there’s a lot of freelance work available out there to game the system), then it’s pretty much exactly the prisoners dilemma playing out in real life – the only winner is Facebook as they are getting revenue, but even then only over the short-term as advertisers can get pissed-off and leave (and are).  All the ad networks take quality of CTR’s seriously because it’s their bread-and-butter, but what can they do?

So, going into the main topic of my post – how do you combat click-fraud?  For the advertiser there’s advice out there on a few things they should do, but for the purposes of web security that I hope will become clear later, let’s just focus on the ad networks themselves in how they can address click-fraud.

1. Firstly, the ad delivery network (what I’ll call the “system” from now on) has to keep stats on what was served and clicked for their clients.  Any unusual spikes or patterns in these stats could cause concern (could because their may still be many legitimate reasons) and need to be investigated.  This is no different than security monitoring and historically reviewing log files to look for anomalous behavior.

2. Ads are usually targeted at users and are supposed to be in some way randomized so the user doesn’t see the same ad all the time, and multiple clicks on the same ad shouldn’t be counted if it’s from the same person.  Mostly, targeting is done geographically and/or based on characteristics of the user – age, interests, demographic, etc.  In order to access the ads the fraudsters have to be able to come off as “real” users that match those characteristics (and enough real users so that it doesn’t trigger the abnormal pattern detection, or the “real vs bot” detection that I’ll talk about below) so as to enact the clicks.  The geographic part is pretty easy – there are shared proxies and VPN solutions, public and private, that allow you to look like you are in any country you like (I use such a service sometimes to watch the BBC iPlayer and catch up on shows at home).  Obtaining legitimate users matching a demographic can be achieved either with automated sign-up tools, or simply paying low-cost labor.  This is very much the CAPTCHA arms-race we are going through now, and there’s even systems out there that will “nourish” and “grow” an account, much like a real user would add pictures, wall posts, etc, so it’s not as obvious that it’s an account used just for click-fraud and face being shut-down at some point if it’s discovered.

3. To the people gaming the system, the risk/reward ratio currently is very much in their favor, whereas the risk/reward ratio of, say, robbing a back is low – there’s a good chance of getting caught, banks often don’t lose that much money in such an incident, and the punishment if the robber does get caught is high.  In order to re-balance the risk/reward ratio, many companies are going after the perpetrators click-fraud, much like they have been know to go after spammers or hackers.  This is done in the hope that it drives the perpetrators away from the system that is going after them, or the activity in general.

4. Lastly, the system has to make a determination of what is a “real” user clicking on an ad vs. some automated program (bot).  There’s various ways this could be done – IP address, speed of viewing/clicking, browser version, referer header, cookies, etc, etc – and the method(s) are kept very close to the chest because they could all be spoofed and gamed, especially if you know what the rules are.  The corollary with webappsec here is that we have very much the same issue with session management and session hijacking – we’ve had to load something on top of HTTP (cookies) to provide state and user identification which may be spoofed so we might not have any idea who the “real” user is.  HTTP by design doesn’t support the level of user identification that would really help and I very much doubt that it ever will (based on the backlash against tracking cookies and per-machine IDs – client-side certifications would be a similar approach, although like with Intel, there’s big privacy issues to overcome), but it sure would be helpful to identify users uniquely on the web. 

So, I think that click-fraud has many parallels with web security and has some really big shared interests. However, I don’t think advertising click-throughts are ever going to be that reliable and will always have this fraud arms-race going on.  Really, advertisers aren’t interested in their ads being clicked, but what people do after they have seen them, and why Cost Per Action (CPA) is where things are heading.  Sure, there’s going to be fraud attempted there as well, but the economics are a lot more solid when you’re paying for an end-to-end transaction with a user rather than paying them just to visit some webpage.

Now, he puts out a good argument – there’s certainly times when it’s much easier to enter a password (especially on a mobile device) when you can see the characters you are entering.  Also, knowing you are entering in the right characters probably will mean less errors and people using stronger, more complex, passwords.  There’s also times when you know you want to protect from shoulder-surfing, and a simple checkbox could re-enable password masking on the few occasions it’s really required. All very valid points.

My worry is though that without the “type=password” attribute, browsers won’t know that it’s a password field and won’t protect it accordingly.  All browsers cache data, depending on settings, to “help” users when they have to re-enter information, but certain data should always be discouraged from being cached – it’s just too easy to dig it back out of the browser [PDF link].

Liberal use of the “autocomplete=off” attribute on these forms (or, non-standard[ly], on the fields themselves) does discourage browsers from caching that data, but in my trawls around the web both as a user and a security tester I see it used very infrequently.

Having the “type=password” is a way of “default denying” as all web developers know to use this field for passwords and the browser just takes care of the rest by not caching these types of fields, and even providing a warning if you are going to send them in clear text over HTTP.  Developers don’t even have to think about it – they know it’s a password, so a password field they use, and leave the browser to provide sensible default protection.

Unless we are going to change the way that browsers render “type=password” fields, and leave the web developers to use them as intended, but with subtly different UI (opt-in of course), I have to say this is a non-starter for me.  I certainly don’t want to see developers “faking it” with JS/AJAX and modifying input box types on-the-fly because once again that can introduce insecure behavior if you’re not really careful.  We have a default “fail closed” state in password fields and modifying it, calling for it to be changed, or just plain removing it I feel will introduce issues because there’s already a built-up expectation of the working-model – developers don’t change their working model or practices all that easily.

Oh, and the other point Jakob makes about reset buttons – I fully agree – kill the buggers buttons.  Thing is though that I rarely see them any more so I guess we are getting a little better (in UI at least).

UPDATE: Looks like this has hit Schneier’s site, and as he gets obviosuly gets more viewers that I do, it’s good to look at the comments.  Most people seem to agree in leaving the password field alone.

On the chat today, one of my colleagues is testing a (traditional client-server) app and it turns out that many of the inputs/output are encoded.  We share our favorite encoding/decoding tools, but a through springs to my mind – what if everything, absolutely everything, inputs, outputs, cookies, etc, etc, were encoded, in an app and not just in some straightforward way like base64.  Perhaps in the programmer’s own "custom" encoding.  From black-box testing, how easily could this be tested?

Tools would simply fail; unless you owned the tool, or knew it well (enough to tweak it, what can/can’t work), their "dictionary" of tests just wouldn’t work – each "attack" would have to be encoded/decoded either in place or on the fly to be understood by the server (/app) or client (/browser).

Manual testers would often just give up – the overhead of figuring everything out, crafting various attacks, running them, modifying them, etc, etc, could just get too large to manage.  Most testing (either pen tests or QA) is time-boxed anyway, so would this "encode everything" route simply run out the clock.  Things that a tester might get round to looking at, or identify as needed to be tested could just get missed. (I appreciate that this is just as importantly a scoping issue for you contract/billable-hours people out there, but just run with this for now ok )

Would attackers have the same problems?  Unless it’s a big, juicy target (like Google/Microsoft for example), would the investment pay off for the pain? Would they also give up and look for easier targets?  Does, in this case, security by obscurity pay off to some degree?  Remember we are not talking encryption here, encoding only.

Now, let’s consider the same app from a white-box approach.  I posit that it’s *much* easier to test such an app.  Once data (inputs/outputs) are inside the code, it’s easier to track how they are being used.  Even if they are kept in their encoded state until the very last minute (or not even decoded at all), it’s much easier to spot code constructs that are "bad".  Think, as a simple example, SQL injection – seeing a query being created on-the-fly may not actually be SQL inject-able, but it certainly is a cue to look into it a bit deeper.  Same with many of the other vulnerabilities.

Perhaps I’m even evil enough to modify one of the hacme apps, or some of the other code I have lying around, to be such an "encode everything", and submit it to some people and tools for testing.  Anyone care to guess what the outcome would be?  Perhaps even engage in this little experiment themselves?

I don’t believe that black-box testing will go away – it’s too engrained into our industry/culture, and it gives a good bang-for-the-buck if used correctly.  However, I think that this is at least a good argument for grey-box testing.  Having the code available to a tester, even in a non-compileable state, just to look at when necessary, would help immensely in these "what the f is this this app doing" situations, and quickly identify some problem areas to look are more deeply rather than going down the preverbal rabbit-hole.

Bug reporting

Everyone knows how to report a bug right?  Repro steps, symptoms, etc, etc.  Just as everyone knows how to fix them – reproduce, find the root cause, fix it, test the fix.  However, there’s two parts on both sides of the equation (test/dev) that I’ve often seen left out.  Testers don’t show enough evidence so devs know it’s and issue to fix (so mark as non-repro), and developers don’t go back into the rest of the code to find/fix similar problems (they just fix the issue they have been provided).

This happens a few time a year with me – I’ve found some vuln (say XSS) that is systemic in the application, but it’s pushed back as not an issue or non-repro, or if it does get fixed it’s only with the specific test case I supplied.  So, here’s my recommendations in dealing with that.

For every (well, non trivial at least) high-risk vulnerability I discover when doing a pen test, I do a screen recording of it.  That video helps in a number of ways.  Firstly, it specifically details the reproduction steps in a way that can’t be misconstrued (and I do a voice-over explaining why I’m doing it), and also it show the issue in a way that it can’t be dismissed as non-repro – the dev/PM can see it happening on at least my machine, so it’s valid in at least one situation/environment and worthy of investigation if it’s not the same in the developers.

I’ve used Techsmith’s Camtasia product for screen captures from way back when I had a copy at FloridaTech.  It’s not cheap, and I should really upgrade it now, but even the older version works well enough for me for what I need it to do.  If cost is an issue to you, try Microsoft’s Windows Media Encoder – it’s not as simple, but will still get the job done.

Make the decision yourself – would you rather watch a vuln or read a description like the following

On the "buy a book" page, the quantity drop down isn’t validated and can be set to a negative integer.  Doing so, an attacker is able to purchase a negative quantity of books, and therefore obtain a refund on their credit card.

What is possible with this description is that it’s lacking reproduction steps – the developer could look at the dropdown and just say that a negative number can’t be chosen.  Bug closed – no repro.

Although generally it’s not the testers role, but when Foundstone logs a finding we always provide a detailed recommendation for each issue discovered.  This gives the developer an idea of how to fix it in a way that isn’t just to patch that one specific occurrence.  That way you encourage "correct" behavior (i.e. centralized validation, consistent access checks, etc), and for the developer to go off looking elsewhere for similar issues.  Doing this for black-box testing is sometimes difficult (you lack the context), but it at least provides the developer somewhere to start.  In saying this however, developers clearly have more insight into the inner workings, know the app better, and might be really experienced in what to do, so the recommendation provided may not be acted upon in the exact way, or perhaps ignored completly for another approach (including doing nothing and just accepting the risk) which leads me nicely on to methodologies.

Methodologies are not for idiots

One of the things in Joel’s post that I didn’t agree with is that methodologies turn people into compliance monkeys.  At Foundstone we have methodologies for most (not all) of the service lines we deliver, and there’s a reason for that – without a methodology to "guide" a tester (even an experienced one), it’s very easy to go "off down rabbit holes" chasing a possibly promising bug, and end up with little time to test all the things we are supposed to look at on an engagement.  Methodologies keep people on track (even if it is looking at "best practices" a post on that little topic soon!), provide a means for a baseline of activities, and (as Joel points out) can be used as to ensure a repeatable process.

However, in saying that, there are some issues with methodologies that are pointed out.  For experienced testers, the last thing they want to do it get caught in the weeds – detailed descriptions of what to look for/do are fine for inexperienced tester where the guidance helps them, but just get in the way of experts, who really just need a checklist of things to do without the "fluff".  Also, there has to be room (and encouragement) to go off the methodology – no test plan is totally complete/sufficient, so ad-hoc testing in areas that perhaps aren’t in a methodology in important as well.  When I’m writing a methodology at Foundstone, my "test cases" (for want of a better term) look like this.

Perform TCP sweep on the target server.  Note any extraneous open ports.  Optionally add findings for any open ports other than 80 and 443 if external test.  Be aware of duplicate findings if importing Foundstone Enterprise scan

****************************************************************
Why:
Web servers should only communicate on two ports – 80 and 443.  Any additional ports open to the outside world expose services that may reveal a greater attack surface of the server/application than is necessary (much of the "internal" services a web server may require for maintenance/operation such as SSH, SMTP, etc should be firewalled off).  Most modern web servers also have administrative ports that should not be available remotly, but sometimes due to misconfiguration are – allowing access to these, even if authentication is required may be considerable risk to the server.

How:
Run a full TCP port scan with Foundstone Enterprise/NMAP.  Note: this is often not the default behavior for any of these tools. check the settings of the tool you are using. For TCP source port scan, ports 20, 53, 67, 80, and 88 are most likely to give fruitful results.

1. Foundstone Enterprise set the TCP Scanning dropdown in the services settings to "all"
2. nMap use -sT -p1-65535 (or -sS -p- if you just want a "quick" scan, but be aware of the limitations)
3. Under some circumstances, changing the source port will allow for connection through a firewall.  Use the -g flag in nMap to do this.

For more info on nMap options, see http://nmap.org/man/

The sections increase in detail so that the tester can get the level of info they need.  An experienced tester just reads the introduction so they can remember what this test is for and they know to stop reading when they see the asterisks.  Perhaps I need refreshing on why I’m doing this test or what to look for – for this I look at the "why" section.  Finally, if I’m a novice tester and don’t know how to perform the test case the "how" section details it in steps (including tool commands/parameters if appropriate, although these are auto-configured in the tool I’ve written to help guide testing so the user should just be able to hit "play").  Different sections of the methodology use information gathered from previous tests, or can be used to "confirm" valid results and test completeness.

In any case, I’m glad that I came across that old post from Joel, and really like reading his insightful material both online, in Inc magazine, and the stuff on StackOverflow, so please add them to your RSS feeds.

Its a great article, and Gary is spot on, but I had a couple of points I wanted to discuss, so I emailed them off.  Thankfully, Gary is a friend, and is really good at arguing any points I might have.  For interest, here’s some of that email exchange.

MA: CSRF I don’t really consider to be an interposition attack – the attacker isn’t really getting "in between" anything, but causing something to happen (with trust) that otherwise wouldnt. In many ways I think this is more of a replay/reuse attack.
GM: I think it is a classic interposition. In this case, the browser is being treated AS the user, which is incorrect and leads to the vuln.

I sort of agree in that the browser is being treated as the user in some way, but I still think of this attack as a form of "replay" attack (perhaps even "reflected replay", as with the SMB vuln patched recently).  Interposition is, according to the dictionary, something that is interposed, which is defined as "in between" (which CSRF is not – the attack is an initiator. In any case, I think this is just semantics.  Would be interested in what other people think this class of attack is.

MA: CSRF isnt "always" in play when XSS is present – on .NET for example, the viewstate has a MAC created by the server on each request. If you add random data (for example) the session ID to the viewstate (various ways of doing that) the attacker can’t create/send the viewstate (as they can’t hash/re-sign it), and therefore the server discards the request. That’s on the .NET platform – I dont think I know of any other platforms/technologies like that, but there might be. There’s lots of "but if’s" in there, but that’s quite a "normal" case.
GM: Thanks for this example. Didn’t know of it prior. I stole the thinking about the one implies the other from Felten (see the link off the SB page)

It’s really common that any protection measures put in place against CSRF totally fail when XSS is present.  There’s one case that I know of though where this isn’t totally true and that’s in .NET.  The reason is when the viewstate is used, it’s value has a MAC, and any tampering of parameters can be detected by the server.  On it’s own this still doesn’t protect from CSRF, but if "random" data is added in (preferably with each request), then even if the attacker has XSS on the page and can read the anti-CSRF token (the viewstate), they can’t reuse it in another request because it’s a) changed after some interval and b) they can’t send a valid viewstate (re-created or modified by an attacker) because they can’t "sign" it (add the checksum) as they don’t know the server-side secret.  If the viewstate doesn’t match what the server expects, it throws an error.

It’s certainly an edge case, but one I hope that other anti-CSRF systems consider.  Alex Smolen (colleague, ASP.NET guru and MSFT MVP) knows much more about this, the issues, and has posted about them here.

MA: Web vulns / web security have a lot of attention for a few reasons I think.
  – it’s easier than "traditional" security. you can test without have to set a lot of stuff up, and the protocol/tools are very simple to write/use
  – it’s not rocket science. Jer/RSnake are certainly pushing the state-of-the-art, but a lot of it is still down to server (input) validation and browser insecurity. I’ve not seen a "new" attack in the web space for a number of years now
  – Lots of software is going onto the web. I read somewhere that it was the most dominant dev platform (in terms of number of people programming for it). *All* of that software is in one of the most hostile places to be (accessible remotly, often containing some level of assets), so people focus on it
  – it’s a very fertile ground. WebAppPenTesting is still a "fish in a bucket" exercise. Devs are *still* getting basic things like XSS wrong, although I dont like just searching for vulns – as you mention, you should be looking for architectural issues and the root causes of vulns, as otherwise it’s just "penetrate and patch" and the client doesn’t really learn anything

Lots of acknowledgments between myself and Gary here, so I won’t bore anyway with the details.  There is the challenge for me to actually find the reference of the web being the most dominant dev platform, which I’ll have to look for.  Anecdotally though, by far the most software that the security consultancies I know are reviewing/pen-testing is web-based, although that my have a self-selecting bias (because of risk/exposure).  I don’t have any hard figures, but I (and Gary) would certainly be interested to see them if anyone has them.  Otherwise I’ll have to rack my brain in where I saw that (and also hope that it wasn’t just in some marketing literature!)

One thing that Gary did put me straight on was the following

MA: The tools that are out there are just useless. Even more useless than the code review tools are, and we know what it takes to get them running and making an impact. The issue is that each and every webapp is custom, and although there’s a nice simple protocol, actually recognizing when something went "wrong" (discovered a vuln, got logged out, not discovered all the pages, etc, etc) is actually very hard. Jer is being generous in saying that a tool can only find 50% of the issues. Press a little more, and that is "a tool that is well configured, knows the app well, and has a good idea of the vuln and it’s external characteristics" – the stars really do have to align to get even close to that 50%. We really need tools though as this issue it’s scalable to the number of people we have looking it.
GM: I disagree with this one. The tools find lots of "stupid" things you should have known better about. That is a good thing. How much can QA people do as a smoke test?

I think that this falls into my last sentence.  I really do believe that we need tools, as otherwise it’s just not scalable, and as Gary says, how often can QA people look at something to give it "a quick test" for certain issues.  That’s why I think the "badness-ometer" quote is so good and great advice.  In general I would like to see these tools catch up, and for some level of acknowledgement (other than from a few people) in that they will never replace a human for a large part of the testing.  This really has to start to change, and I have thoughts on that (which I’ve shared with various people), but save them for another post.

My final point was

MA: the final thing about web security (both tools, practitioners, and research) is that it’s much more often than not done black-box. This "stabbing in the dark" is only going to find so much, and even then only some things. There’s still this pervasive attitude out there that pen testing, even webapps, is enough. Im amazed that we (still) find as much as we do, which is a sorry state for the webappsec field.

I’m certainly guilty of my work being based around black-box techniques, but that seems to be what clients want and are willing to pay for.  There’s few-and-far-between clients that want to go "deeper", and often that’s because (IMHO) they don’t see a ROI/benefit in going any further.  Black-box is a good place to start, but I’ve always considered it just one stage in a multi-pronged approach.  However, one has to realize that the majority of companies out there do not make their living of their software – it’s an "enabler" for them to do business, and therefore it only has to be "good enough" – any more and (in some way) they are wasting their money.  You don’t have to outrun the bear, you only have to outrun the last person.

In any case, I really like having discussions with Gary (even thought the roles were reversed this time), and can’t wait until I have another opportunity to sit down by his river, chat, drink, and watch the world (and black hawk helicopters) go by.

The first one, Credit for Researchers, I think is very important.  From my academic days, referencing previous work was de-rigeur and you just weren’t taken seriously if you published or spoke without noting the people that laid the ground before you, and other views/options on the subject – basically not referencing was a way of admitting that you hadn’t looked at what came before, or other ongoing work in the area, which was "bad" (ignorant to be more precise). 

There’s lots of security research going on out there – independent or part of an organization – and from what I’ve been seeing for the past few years, very few of it is totally original.  That’s not saying that this is not ok, as it’s to be expected (encouraged in fact), but I’m seeing very few people reference previous work as part of a disclosure.  This means that either a) these people are forgetting to reference previous work, b) they are not referencing on purpose (as it makes their work look more "original"), or c) there’s a lot of "independent re-discovery" going on.

I’m not sure which it is, but there’s issues in all three.  Re-discovery is a problem because that means that we aren’t learning from previous knowledge.  Not referencing when you know of other work is academically dishonest.  "Forgetting" (or not being aware of other work) is ignorant and only focusing on a small part of the problem (edit to add: perhaps you did find a "unique" vuln/issue – congrats, give yourself a pat on the back, note how unique your finding is, and now go back and reference places where it should have been looked-at/discovered/thought-about, and perhaps why not).  I call, as does Chris in the Veracode blog post, is for us to get a bit more academic in the way we approach security research.  If not, we’ll be doomed to this never ending hamster wheel of pain feedback loop, making the same mistakes / finding the same issues / never addressing the "real" issues.

The other post (also from Chris Wysopal, also from the Veracode blog) is one saying we’ve reached the application security tipping point.  I actually think we reached that tipping point some time ago (2 years?) where less and less vulns were being discovered in OS and large vendors, but increasingly in software from thousands of smaller software vendors. 

I’ve spoken to a few people about this, and I don’t think that the overall security future looks all that rosy.  Microsoft are spending a lot of effort on security, and it seems to be paying off (or not).  When MSFT manages to lock down their software so that the effort in finding vulns is not worth the payoff (mitigating controls, defense in depth, quick discovery/response/resolution, etc), where is the attention going to go?  People I’ve talked to at MSFT say that their job isn’t over by a long-shot; they are very much interested in the full ecosystem so helping users create secure products is also in scope, but once again, where do the hackers go? 

They don’t just go away, they go to the next level of lowest hanging fruit.  It might be other vendors (Apple, Adobe, Google for example) which may not have the focus that Microsoft has been forced to have, or even worse, smaller players like custom websites or things like Wordpress, Movable Type, phpbb, vbulletin, etc – software that has a huge install base, but perhaps not the resources to deal with a full-frontal attack.

In any case, unless lots more people take these issues seriously (and to their credit, many are), I don’t necessarily like the way this could be heading.  Rather than having a few "threat sinks", we could end up with many, and down the path of very custom attacks that are incredibly hard to detect and protect against.  It’s certainly in progress in the cybercrime community (see Iftach "Ian" Amit’s BlueHat talk if/when it becomes available) and could only be a matter of time before it’s widespread. 

Think of it security’s own Hydra – cut of one head (vulns in a major vendor), two grow back (vulns in smaller vendors), and that’s a worrying proposition.

Just before BlackHat we had the whole "DNS is broken" fiasco.  Not knocking Dan for his research and discovery of the problem – sure, as many people say, the underlying flaw was at least identified a long time ago – but what what he did was make the underlying vulnerability into a workable exploit and kudos for that.

Next up we have the TCP flaw which if we are led to believe can bring the internet to it’s knees with the use of just a simple tool.  Once again there’s speculation that this isn’t a new issue, but something quite old brought new again.

Recently we have "clickjacking" which, once again, the industry press are pushing out to be the next major threat.  I’m a little closer to this field than the others (networking not really being my "thing"), so when Jeremiah and Robert’s talk at OWASP was pulled (thus raising the interest level) I had a look and immediately understood what they were going on about.  Again, it’s not brand-new – there’s been talk about iframe injection and click stealing in the past – but these guys have improved the exploitability of the flaw and are raising awareness.

I’m not out to knock any of these guys doing this work. I think it’s great that they are researching into security issues (and yes, I am a little jealous that they get the time to look into these things) and even more happier that they are getting the message out.  At the very least in the case of clickjacking RSnake had been very clear in where the previous work and the impact (they didn’t pull their talk BTW – they were asked to), and Dan had to walk a very fine (and high) tightrope in order to get more people to patch,  but there’s very little "sexyness" for journalists (or blog authors for that matter) in writing anything level-headed.  When these news articles go out CISO’s come round, article in hand, asking what they can do  and getting back blank looks – no-one knows what the issues are, the drawbacks/side-effects/risks of patching vs not patching, or where to go for more information/advice.  This just leaves both sides frustrated.

There’s lots of smarter people than me adding their $0.02 worth (including the researchers themselves), but "responsible disclosure", especially when it’s associated with someone revealing details about the issue at a conference is turning out to be more like "partial disclosure" as other smart people play their own 20 questions game and in turn figure out at the very least a rough approximation of the issue.  It is right to do the whole responsible disclosure thing and work with vendors to get things fixed before it’s common knowledge but saying "I know something you don’t know" just makes other people want to know that information to, and our industry is full of bright people that given the motivation (which very may well be showing the rest of the world how bright they are, or stealing the limelight, not to mention the black-hats out there) are plenty capable of working it all out.

I guess what I’m asking for is if you do find a vulnerability, by all means work with a vendor to get it fixed, but (and I know this can be very difficult with the fame and fortune just around the corner) using it as a "PR" event for you/your company or scheduling the big reveal with a major conference all the while holding the details back I feel is a recipe for disaster partial disclosure.  If too many people go down this route, and it appears that it’s becoming more and more common, it’s going to end up like the boy who cried wolf – when there is a major vulnerability everyone is going to be so de-sensitized they just aren’t going to care.

Usually, I like competition, especially in the technology marketplace – the more people offering products/technology, (generally) the better those things have to be to survive and gain users – modern day Darwinism.  However, in this case I’m quite apposed to another browser out there for various reasons.

I didn’t know this proverb until Jeremiah shared it, but it makes sense…

"There is a proverb that illustrates the way to quickly determine whether or not someone is sane. The individual is shown a river flowing into a pond. He is given a bucket and asked to drain the pond. If he walks to the stream to dam the inflow into the pond he will be considered sane. If, instead, he decides to empty the pond with his bucket without first stopping the in-flow then he would be considered insane."

If I can use the analogy, what we have here in Google Chrome is adding more water into the pond we are trying to drain.  The water, if you want to follow the analogy, are the vulnerabilities in web browsers, badly written or just plain malicious websites that users are trying to protect themselves from (often via plug-ins), and the usual issues with web development we all have when trying to get a site to "just work" simply with all the current versions of browsers that are already out there.  With Internet Explorer and Firefox slugging it out, Safari and Opera distant runners up, we were sort of, slowly, gaining ground on addressing the issues.  Between IE and FF, lots of good security work was being done making these browsers more resilient and we were getting a good handle on the overall problem.  An additional browser, with what one expects to be a growing market share just because "it’s Google", puts a hole in the dam and water is filling up the pond again.

I understand the monopoly argument, and buy why it’s not always such a good thing, but there are exceptions.  Office/Word is one exception – the ubiquity of Microsoft Word, either the product itself, or that many other bits of software can read it’s file format, is good – I can send a document to someone and know with a very high probability that they can do something with it.  With two (plus change) browsers out there, we know what "issues" each have and ways to mitigate/work around them.  Granted, Chrome is based on a common and open-source rendering engine (same one used in Safari), but it’s another platform that a site or security issues need to be tested on.

So that’s one point.  The other point is that it seems that all the mistakes that other browser vendors have been through haven’t been learnt in the development of Chrome.  In the first few weeks, numerous vulnerabilities were reported (which should be listed here, but I don’t see a "security" tag, so this list will have to suffice), many of which have been previous issues with IE/FF.  I think this is common with Google, as IMHO they still have a "start up" mindset instead of one of a mature software company where the quality of what they put out really matters.  Thankfully, I’m not the first to say this – David LeBlanc says pretty much the same, RSnake takes it further.  Rushing to market is one thing, but it’s totally different when you make more work for people (and potentially the web a more dangerous place to those not expecting it).

The flip side is that it’s "beta software", and releasing it now exposes it to the world so we can tear into it, finding the bugs/issues/vulnerabilities, and the software gets better.  That’s good – it’s a common way of releasing software.  However, track records need to speak for themselves where almost half of all Google products are (still) in beta.  Until something becomes a full product (and often even-numbered versions – apparently it works for films as well) they shouldn’t be considered "fit for everyday use".

The main reason for these problems though is I feel an underlying reason that many people forget about Google – they aren’t a software company, but an advertising company – pretty much all of the money they make is via pushing out ads, which has very little relevance to the quality of their software.  That’s not to say that the guys writing and testing the software there really don’t care – I know that they do after being invited up to do a talk – but it has very little relevance to the bottom line.  Just ask Microsoft on how even the perception of a buggy/bloated bit of software can affect them – it slows down sales and adoption.  Microsoft’s money is made off software (and predominantly just a few titles) so they have to be good at that.  I can’t help feeling that in Google’s case their software is a side distraction – a stepping stone to another goal.  As long as people are still hitting their search engine and embedding ads with adsense, I fear the worst kind of "good enough" software development, as evidenced to some degree by the continued "beta" status.

I really try to like Google, and have even toyed with joining them a few times – the people there are really smart, some of the most friendly guys in the industry, and they have huge reach with interesting problems to address.  I also thing that the web is the operating platform of the future, and therefore all the software/technology they are developing is the way to go.  With Chrome they have architected security principals into the system which will I’m sure pay off later, and it’s clearly a technology base for future things.   When it comes down to it though their objective is gathering information from as many different places as they can and using it for their main business purpose – pushing ad’s.  Finding things in the EULA (even if by mistake), and a potential prospect to use the browser to access even more of the web (although the toolbar does much the same job), doesn’t make me feel all that comfortable and instead reaching for my tin-foil hat.

Perhaps I’m being way too harsh here (Matt Cutts gives his take here), but I feel that this is a step backwards for web security (in the short-term at least) more than anything else.

Security is much more than finding/fixing defects

The first post I saw on this was from Ivan Ristic, where he says…

Underneath all our security issues lies our inability to write defect-free code. Solve that and we’ve solved the security issues. Focus on the security alone and we won’t solve anything.

This sort of follows from a really old post on the Microsoft SDL blog from James Whittaker – Reliability Vs. Security.  Defects are defects, functional or security, and we’ve been working on the former for some time.  Our techniques are better, and we’ve learnt a lot, but we still can’t write error free code, and there’s a lot of prior art/knowledge in the functional side that the security side has to catch up with.  One of my favorite (old) quotes is from Chris Mason, ex-development manger for MS-Word, who said “Since human beings themselves are not fully debugged yet, there will be bugs in your code no matter what you do”.  This leads me to the most recent post on the topic that I saw (also on the SDL blog) from Michael Howard.

Security is bigger than finding and fixing bugs

This isn’t any deep insight or rocket science, but it seems that some people have to be told it anyway.  Yes, currently there’s a lot of security issues being found after products are released, and they often track back to implementation issues (leaving aside for the moment if there were any security requirements), so people think if you patch the issues and/or look at the code then that’s 80% of the work.  This might be true for now, but so much of these things could get caught at the design or even implementation stage.  Also, I would bet that over time implementation flaws are going to start to go away as we code up defenses in frameworks, languages, API’s, etc, much like buffer overflows are much less of a worry in managed code.  Even if we get to that point there will still be vulnerable system out there because of poor design decisions, failed logic, and other problems that are simply not associated with the code at all.  Focusing on the code (and therefore testing/fixing just the code) is IMO the wrong approach.

MBTA vs MIT Students

Talking about flawed design, here’s a good example that I’ve seen lots of posts about.  The best intro is from Chris at Veracode here and here.  Basically, the Massachusetts Bay Transportation Authority (MBTA) is (was?) suing some students to stop them from presenting about a flaw that they discovered in their payment card system (turns out many other transit authorities the world over also use the same system, and thus vulnerable the same way).

I first can’t believe how basic the vulnerabilities are (which surely would have been caught in requirements/design than the much-more-expensive delivered system – once again, testing/code is not the answer), and second that anyone would defend what the MBTA is doing in suing the students (even though I understand it’s “risk prevention” on their part – this is not the right place to do that).

This leads me to something that has been going on for some time in the functional world, and perhaps needs to from a security aspect as well – software warrantees.  When someone purchases a physical item there’s an expectation of “fitness for purpose” – if it doesn’t work you can take it back for a refund, and if it blows up in your face (during “normal” use) you can sue the manufacture.

It’s extremely rare to find any warranty for software products, and the EULA is often the “get out of jail” card for the implied warranty.  I’m not sure that we’ll ever see true software warrantees because of push-back from software producers and the additional cost it would (could?) add.  However, having enforceable software warrantees would both force vendors to ensure that their software is “fit for purpose” (which in the MBTA case it clearly isn’t) and allow the purchasers of said software some rectification – I’ve had my fair share of clients that I’ve worked with that have had real problems in getting fixes to vulnerabilities I’ve found in fixing them in a timely manner because they don’t really have to.  For some purchasers (government(s) being a good example as Schneier points out) forcing these (because of their purchasing power) would force an improvement in quality, both functional and security, that would benefit everyone even if it’s only in one version/variant that you could buy if you wanted that level of expectation.

In any case, I’d like to know anyone’s thoughts on this.

(As a further resource, badsoftare.com is a site an old colleague of mine put together a while back that might have some interesting information for people.)

While onsite with the client we went to a users desktop and was doing some things when the user popped back and was watching us work.  He was fine with it and all – a good communication had gone out around the company explaining what was going on, the systems that were being shut down, and allowing us access to whatever we needed (I can’t tell you how rare that is – many companies continue to try and operate as “business as usual”, but this one really did come to terms quickly and take the appropriate action – kudos to them for that).  However, in introducing me to the user the client IT person simply said “this is Mr. Wolf – he solves problems”.

The quote obviously is from Pulp Fiction, and got me thinking on how apt that introduction was.  When on incident response engagements it’s rare that when I, or one of the other Foundstone guys, get called in we have specific skills that the client’s IT staff do not have – after all, they are the ones that look after the systems day in, day out, during normal usage.  What we do bring though is a cool head, an assessment of the situation from previous experience which leads to a plan, very good general knowledge about all the systems/technology/thing going on and how they affect the current environment/situation, and most importantly contacts.

The cool head is important – often the local guys may be very stressed out (it’s their systems under attack after all) and oftentimes have been working long hours trying to address the problem before they have called us and we are onsite.  The plan is equally important because otherwise people are running around doing “things” which may not be productive at this very moment and there’s no idea of progress.  But the key is access to contact that are very highly specialized in particular areas.

It would be really nice to be an expert in everything, but with today’s computing technologies there’s just far to much for any one person to know.  I may be an expert in the web and web application software, and it’s useful for me to be put on those kinds of IR engagements where possible.  I can also reverse engineer viruses, look at SQL databases, understand WireShark traces, look at Solaris boxes, etc, if necessary, but I’m not as good as people who do this every day and have labs setup to work any issues in these environments (on site it’s usually me, a laptop, and sometimes some additional hard-disks or other “gear” to capture what is going to be useful later).

So I thought that was a really insightful analogy (and thanks to that person – you know who you are).  Mr Wolf doesn’t necessarily have any skills that Jules and Vincent don’t have, and the actions he gets them to do are nothing that they couldn’t have done (or thought of) themselves if they were level-headed.  The single thing that he did have that they probably didn’t is the contact at Monster Joe’s Truck and Tow.

The Foundstone guys already have a new nickname for me, and a little skit [warning, some language in the link some may not appreciate].

And no I don’t dispose of body parts.