This is the story of a Manila security hole. Manila is a server-side content management system by Userland, the same company that makes the client-side CMS/aggregator/weblog tool called Radio. I want to be clear that this Manila security hole was closed on January 29, 2002, and the fix was immediately deployed to all the major Manila-based weblog hosting sites. So this is nothing to get panicked about if you have a hosted Manila weblog. Also, as far as I know, this bug does not affect Radio users.
The Manila security hole, in brief, was that referers were not HTML-escaped before being listed on the referers page (usually /stats/referers). This allows malicious clients to alter their referer string to include HTML tags, including Javascript (in a SCRIPT tag). There are many potential problems with this: hackers could insert web bugs in your referer page that track your visitors, or they could open popup (or popunder) windows with advertisements that appeared to be initiated by your site, or they could redirect a visitor to another page altogether, perhaps one that took advantage of other security holes in the visitor’s browser. I would expect any competent software company to be able to recognize these potential problems, once they were advised on the underlying code bug.
(Update, 6 PM: apparently the original version of this story triggered a completely different bug in Radio which — and I am not making this up — redirected readers to a different page. This was never my intention, and I’ve tweaked the story so it does not trigger this bug in Radio. Read Radio escaping bugs for all the gory technical details on that bug.)
I want to be clear that I am an advocate of full disclosure of security holes; however, I also believe that product vendors should be given a reasonable chance to fix their products. This has always been (and will continue to be) a delicate balance, based more on courtesy and reputation than science and statistics. How long is a “reasonable chance”? I would say no more than 48 hours — and yes, I’ve worked on complex production systems, so I know how short 48 hours can be. The key here is the realization that, by the time someone tells you about a security flaw, someone else is almost certainly already exploiting it. Even if the person who tells you discovered it independently (as I did in this case), you must assume the worst, that others have already discovered it and not told you.
And what about vendors (like Microsoft) that actively oppose full disclosure, go out of their way to belittle security experts who expose security flaws, and have developed a reputation for ignoring private notification of such flaws? What about flaws that are easy to exploit but difficult to fix? This muddies the waters even further, and I would leave it up to the discretion of the person who found the flaw. I did not have these problems in this case. When I found this security hole in Manila, I was on good terms with Dave Winer (the CEO of Userland); the flaw was both easy to understand and easy to fix; and I had seen instances of Dave openly admitting — and explaining — security flaws in his own products (after he fixed them, of course).
This story is told in the same way that it unfolds, in a series of emails. All the emails that make up this story are presented here unedited, in their entirety, except that email addresses have been replaced by proper names. Except for minimal comments to keep track of the timeline and people involved, I will let the emails speak for themselves.
I started the story on January 19th, by sending this email to the CEO of Userland, Dave Winer.
Date: Sat, 19 Jan 2002 14:54:00 -0800 (PST)
From: Mark Pilgrim
To: Dave Winer
Subject: Manila security holeOut of respect for you personally, I am telling you this privately. (I reserve the right to speak about it publicly in the future.)
I have discovered a security hole in Manila. The referers page does not HTML-escape the referers it lists. This allows a malicious client to insert arbitrary HTML into the referers page, including Javascript. This could, for instance, be used to hijack the referers page and redirect visitors to another site.
To illustrate this, I hacked my Python urllib.py to send this referer [note: remove the dashes to trigger the bug]:
<s-c-r-i-p-t language=”Javascript”> window.location = ‘http’ + unescape(’%3A%2F%2F’) + ‘diveintomark.weblogger.com’ ;</script>
Then I programmatically hit diveintomark.weblogger.com a few times, and waited a hour until the referer page was rebuilt. Now, going to my referer page redirects to my home page. This could just as easily be a page on a completely different server. (The redirect URL is specified in full by the malicious client.)
http://diveintomark.weblogger.com/stats/referers
HTML-escaping all referers should solve this problem.
Cheers.
-Mark
Dave replied almost immediately.
Date: Sat Jan 19, 2002 06:13:01 PM US/Eastern
From: Dave Winer
To: Mark Pilgrim
Subject: Re: Manila security holeThanks Mark — it being Saturday I’m not sure if we can get a response out of the team. we’ll get on it quickly. Dave
Ten days later (January 29), I tested my own Manila-based weblog and discovered that the problem still existed, and I sent another email to Dave. The rest of the story happened within hours on January 29.
Date: Tue, 29 Jan 2002 07:50:32 -0800 (PST)
From: Mark Pilgrim
To: Dave Winer
Subject: Manila security hole followupDave, whatever happened with that bug I found about the unescaped HTML on Manila referer pages? You said you’d hand it off to development, but then my personal life went haywire and I never got a chance to follow up. Is it fixed, or will it be fixed soon?
Cheers.
-Mark
Again, Dave replied almost immediately.
Date: Tue Jan 29, 2002 11:01:21 AM US/Eastern
From: Dave Winer
To: Mark Pilgrim
Cc: Brent Simmons
Subject: Re: Manila security hole followupYou can raise it as an issue publicly if you like, Brent Simmons is our spokesperson on this, he will answer whatever questions you have. Dave
So I sent an email to Brent Simmons.
Date: Tue, 29 Jan 2002 08:19:24 -0800 (PST)
From: Mark Pilgrim
To: Brent Simmons
Subject: Re: Manila security hole followupHi Brent. Dave told me to contact you about this issue. I don’t know how much you know about the background leading up to this. In short, I found a small but important bug in Manila about 10 days ago and contacted Dave about it. He said he would hand it off to development, and I was wondering what the status was. Let me know if you need technical details, I can dig up my original mail that describes the problem.
Thanks,
Mark
Brent replied within hours.
Date: Tue Jan 29, 2002 12:54:41 PM US/Eastern
From: Brent Simmons
To: Mark Pilgrim
Cc: Dave Winer
Subject: Re: Manila security hole followupHi Mark,
We can certainly make the change you’re talking about.
But remember that people can also post discussion group messages and put JavaScript code in their posts.
What we did was to use strict referer checking so that no JavaScript code could cause you to post to your weblog, flip your home page, or do anything else that could cause either loss of data or unwanted addition of data.
Here’s a page dated 5/10/2000 where we talk about having added the referer checks:
http://inessential.com/clientsidetrojan.html
Note that Radio also has referer checking.
Does that satisfy? Or are you thinking of a more specific exploit that isn’t thwarted by referer checking?
-Brent
I replied to Brent, explaining the bug I had found, which had nothing to do with the (long since fixed) bug he was describing.
Date: Tue Jan 29, 2002 01:33:20 PM US/Eastern
From: Mark Pilgrim
To: Brent Simmons
Subject: Re: Manila security hole followupThat is actually interesting information (it explains why I’ve never been able to edit my Manila site when using Guidescope, which strips referers), but it’s not the bug I’m talking about.
The bug is this: Manila takes the referer information it receives from the client and displays it in the referer page “as is”. So if the client web browser (or scripting environment) is hacked to include HTML tags in its referer tag, these actual tags will be displayed in the referer page. Such tags could include the SCRIPT tag to insert malicious Javascript into the referer page. Such script would be executed by anyone who viewed the referer page (either the owner of the site, or any visitors). It could, for instance, redirect them to another site. Combined with the multitude of security holes for IE (and even some for Netscape), this could be very serious.
To illustrate this point, I have (just now) demonstrated the bug by hacking my referer and hitting several Userland sites, including macros.userland.com. If you go to http://macros.userland.com/stats/referers right now, it will redirect you to http://inessential.com/, because that’s what I told it to do. (I can send you the one line of code I used — in Python — to set up this hack, but you can replicate it in any environment that allows you to specify the referer manually.) Essentially, I have hijacked the referer page and redirected it to a web page of my choosing; the effect lasts 24 hours, since that’s how long it takes for the hacked referer to fall off the bottom of the page.
The solution to this problem is to escape the referers you display on the referer page, so that “<” becomes “<” and “&” becomes “&” and so forth. This is standard practice in displaying any text from an untrusted source; whatever programming language Manila is written in probably has a function to escape HTML like this.
Let me know if you need any more information.
-Mark
Brent replied and promised a fix, but he claimed the bug was not really a big problem.
Date: Tue Jan 29, 2002 01:40:50 PM US/Eastern
From: Brent Simmons
To: Mark Pilgrim
Subject: Re: Manila security hole followupI’ll make that change today. Your server will probably update tonight.
However, I don’t know how you could do anything worse than redirect a browser or cause it to run a Google search or something. But then again, we’ll just lock it down and it won’t be an issue anymore.
-Brent
I replied to Brent and cc’d Dave, explaining why I felt the bug was a big problem.
Date: Tue Jan 29, 2002 03:44:49 PM US/Eastern
From: Mark Pilgrim
To: Brent Simmons
Cc: Dave Winer
Subject: Re: Manila security hole followupThere are any number of security and privacy concerns related to redirects. Just today it was announced that every shipping version of Netscape 6 contains a bug that allows a malicious web site operator to read any cookies from your local browser cache.
http://www.theregister.co.uk/content/56/23865.html
This could be used to facilitate identity theft: stealing someone’s Amazon cookie and shipping goods from their account, stealing someone’s Hotmail cookie and reading their email, or stealing someone’s Passport cookie and potentially gaining access to their credit card information.
There are also numerous IE bugs that allow malicious web site operators to execute arbitrary code on the victim’s local machine once the victim has been redirected to a properly constructed page. This is one of the ways that the NIMDA worm spread so rapidly a few months ago.
The web is only as safe as the sites you choose to visit, and this Manila bug creates a kind of “open relay” that hackers could use to hijack your visitors’ browser and perpetrate web-based attacks.
Thanks for giving it your attention.
-Mark
Brent replied that the code had been fixed and would be deployed soon.
Date: Tue Jan 29, 2002 04:29:14 PM US/Eastern
From: Brent Simmons
To: Mark Pilgrim
Cc: Dave Winer
Subject: Re: Manila security hole followupI made the change — < characters in referers are now converted to < entities on the referers page.
Most Manila servers will pick up the change tonight.
-Brent
§
I am no longer accepting public comments on this post, but you can use this form to contact me privately. (Your message will not be published.)
§
firehose ‧ code ‧ music ‧ planet
© 2001-8 Mark Pilgrim