It's been nearly a year and a half since I blogged about the WebVulnCrawl bot, and slightly less than that since the crawling was completed by Dennis.

I was very eager to see the results, and had pestered Dennis via his blog several times over the months (comments from me and others have now disappeared from his blogspot for some reason). So I was initially surprised and then very interested to see a post from this now unfamiliar blog in my reader - "Long Overdue - The Final Post".

Dennis comments on some of the ridiculous overreactions that his project incited. I received a comment myself in response to my original WebVulnCrawl post that seemed to very much miss the point, so I can only imagine that it must have been pretty stressful for Dennis trying to explain what he was doing. A "highlight" of the FUD pitted against him must be the posting to BugTraq, which Dennis dealt with rather well on his blog at the time.

So. The results? As anticipated, there are indeed lots of misguided people trying to hide sensitive information instead of simply not hosting it publicly in the first place. Presumably some of these people are the same ones that berated Dennis about his project. We can only hope that Dennis did manage to explain things to these folks, and get them to secure at least their own domains. However, judging by the figures, fixing these would just be a drop in the ocean:

I crawled about 90% of all .com domains that existed as of 11/2005 (millions of domains, I don't have an exact number), and received about 20 emails and one phone call

Perhaps Dennis final report would have made some headlines and caught some more media attention? Maybe such attention would reach more misguided webmasters and stop them trying to hide their server logs with robots.txt? Unfortunately we'll never know as Dennis suffered a hard drive failure and had barely begun to scratch the surface of his collected data. Which is a huge shame. I'd be tempted to take up the project myself, were it not for the stress and hassle that it would entail.

If someone else does build WebVulnCrawl v2 one day, I should notice. After all, I've got to keep a careful eye on my logs, because my robots.txt is securing all of my secret passwords from prying eyes :)