Back in May, there was a week where, like in any other week of course, a number of security issues were reported to the Yahoo! Security Team, internally known as "the Paranoids". Some of these reports originated within Yahoo!, while some come from the outside. The latter category of issues receives extra attention, as publicly known exploits must be addressed as soon as possible and require engineers to work around the clock towards resolution, including mandatory code review and post-mortems.
At that time, I had taken the occasion to compare two such externally reported issues in an effort to illustrate to other Yahoos the work done by our team, and how two similar cases can lead to significantly different results. Being freshly bloggered up in this tumblr-thing, it then occured to me that this might be of interest to people outside the company as well, and so I'm re-blogging my internal summary, with a few select bits of information removed/redacted to make it suitable for public consumption.
The two issues at hand were a Cookie Stealing XSS in Pipes (which, quite frankly, is one of the cooler things Yahoo! has) and a CSRF file upload vulnerability in Flickr. The two issues could hardly be any more different -- not so much in the nature of the problem, which, as you can see, boils down to proper input validation in both cases, but rather due to the way they were reported and the resulting work. Let's have a look:
On Wed, 18 May 2011 02:03:25, Krzysztof Kotowicz (@kkotowicz), following responsible disclosure practices, sent a mail that notified us of a CSRF vulnerability that would allow an attacker to silently upload an image on other users' behalf.
A bug was opened at 2011-05-18 03:54 and marked as high priority; at 2011-05-18 09:16:58, 7 hours after the initial report, the bug was updated with a comment that reported a fix in the staging environment with deployment to production "in the next 10 minutes or so".
At 2011-05-18 09:26:42, approximately 10 minutes or so later, the fix was reported to have been deployed into production; at 2011-05-18 10:28:44, a little over 8 hours after the initial report, Krzysztof received an email from us thanking him for the report, notifying him of the successful resolution and asking him for a postal address to send a Thank-you-Flickr-T-Shirt to. (Kudos, Flickr-folks!)
This impressively fast resolution of a publicly known vulnerability went about as smoothly as I could imagine; the blog by the initial reporter clearly shows how this yielded good will for Yahoo!, though the issue was not reported on elsewhere (a patched vulnerability is boring); all's peachy and rainbows.
So, let's look at this other issue then...
Pipes Cookie Stealing XSS
Yahoo! Open Hack Europe took place from Saturday, May 14, 2011 at 8:00 AM - Sunday, May 15, 2011 at 7:00 PM in Bucharest, Romania. A semi-anonymous person referred to as "Pax" demo'd a crack against pipes.yahoo.com. (His account of the event can be found here.)
Raymie Stata reported the issue at 2011-05-15 06:10:03 via email, and a bug was opened at 2011-05-15 09:23. As this disclosure of the bug happened in public, it is not surprising that it made it's way onto the various blogs rather quickly. The cracker did not receive the response he expected or hoped for from us (whichever that may have been), and seems to quickly have become rather disgruntled towards Yahoo!, which is precisely where this story takes a very different turn compared to the Flickr vulnerability.
The fix for the reported issue was quickly deployed to all colos (at 2011-05-16 02:18:20, again within less than 24 hours after the initial report). However, as a cookie stealing vulnerability, this also required the due diligence of sifting through the logs and identifying possibly affected users, a task that gets complicated by the fact that pipes can of course be duplicated/cloned.
Making things worse, "Pax" was now boasting to have found another XSS in "V2" of Pipes, but somehow managed to make himself unpopular with the other cool kids -- rumors started to fly that he lifted his crack from somebody else. Either way, Philip Tellis (at that time still employed by Yahoo!) tried to reach out to the cracker, but was rebuffed. Worse yet, "Pax" announced that he was now selling a Yahoo Self Spread XSS Worm (including, somewhat astoundingly, "support for 6 months"), which "steals cookies from Yahoo users and uses them to authenticate itself in order to send spam to the contacts of the victim." This, of course, was also picked up by various blogs and news organizations (some of which reached out to us for commentary).
So this ment that while we had at this point already addressed one exploitable issue in pipes, we had to suspect another unknown issue to be present and for sale on the market. With the v2 frontend being identical to the v1 frontend, this also ment that there really was no good place to start looking other than whatever the attackers may have accessed, all with the possibility that no actual exploit exists, of course. And so the detective work to investigate the possibility of another cookie stealing XSS vulnerability began. Sifting through user logs, pipe access logs, correlating user identities and pipe identifiers with regular expressions etc. is time consuming and was in parallel done with remediation for the accounts possibly exploited.
In that process, the Pipes frontend got a thorough code review, with numerous previously undiscovered issues noted (and fixed), but the problem remained that there was no knowing whether or not whatever we would unearth would be the one thing the attacker was talking about (or whether or not the attacker did not have anything at all).
Eventually, we believe to have found and fixed the related issues, but it took on an order of magnitude more man-hours than the Flickr issue noted above. In the process, there cropped up difficult questions on how to deal with a disclosure of such a vulnerability at one of our own events (which, I think, could probably have been handled a bit more gracefully), how to interact with openly hostile attackers and 'exploits for sale', how to determine whether whatever we found is what we're looking for and it has ultimately shown how poorly (not just our) web applications handle external input.
Of course in parallel, there were other, similar issues, and every week since has brought more, but by contrasting these two experiences, I felt I learned quite a bit about different aspects of the work of our team.
September 6, 2011