It’s a bit hard to even know where to begin with this one, perhaps at the start and then I’ll try and piece all the bits together as best I can.
As you may already know if you’re familiar with this blog, I run the service Have I been pwned? (HIBP) which allows people to discover where their personal data has been compromised on the web. When a breach hits the public airwaves, I load in the email addresses and those who subscribe to the service (it’s free) get notified of their exposure plus they can search for themselves on the site. The intent is always to tread very carefully and responsibly when it comes to handling this data, for example, how I handled the Ashley Madison breach. The contents of these breaches has potential to do harm to both the organisation which lost the data and the individuals within there so I give great thought to what the responsible approach is in each case.
Every now and then, I get someone contacting me like this:
Hey, approximately 5 months ago, a certain hacker hacked into 000webhost and dumped a 13 million database consisted of name, last name, email and plaintext password
Now this puts me in an awkward position. On the one hand, the data would obviously be a good addition to HIBP and the people impacted would really want to know about it. On the other hand, by no means do I want HIBP to be thought of as a disclosure channel. In fact, what I normally say to anyone sending me this info is that unless it’s been publicly documented somewhere, I don’t want a bar of it.
However, a number of things made this incident a bit unique. Firstly, the guy (and that’s usually a safe assumption when it comes to this sort of thing) had already given me the data and it only took one glance to see that yes, it was indeed plain text passwords. He was also correct in saying it was 13M records, in fact it was a little bit more than that. It was very apparent that if this was legitimate, it was indeed a very serious data breach and one that had the potential to impact a very large number of people. So I did a bit of research.
Firstly, 000webhost is a free hosting service for PHP and MySQL:
I usually like to try and get a quick sense of the security profile of a site an alleged breach comes from just by looking at publicly observable attributes. For example, the fact that the members area login is served insecurely:
That’s a rather serious oversight considering these are credentials used to manage customers’ web assets.
Another quick test was to check Plain Text Offenders and sure enough, they make an appearance (which I later also confirmed for myself):
Another good source of info relating to security implementations is XSSposed and sure enough, they have an entry for 000webhost from just the last few days. The details of the risk aren’t public yet, they’ve got another few weeks before full disclosure.
Looking back at the site itself, here’s what happens when you try and register and there’s a validation exception:
Doesn’t look too bad? Let’s take a look at the URL:
http://www.000webhost.com/order.php?domain=&subdomain=sdjflsdhkfhds&name=asdasdf&email=aaaaa@letthemeatspam.com&pass1=ThisIsMyPassword&pass2=ThisIsMyPassword&aggree=yes&error_multiple=&error_domain=&error_subdomain=&error_name=&error_email=&error_pass=2&error_tos=&error_number=&error_js=&error_disposable=1&error_bad_gmail=
Yes, that’s the credentials in the URL of an HTTP address so now it sits in all sorts of logs, browser history and other places which are both obtainable by anyone the traffic passes through and by anyone with access to any of those logs.
A little searching the Twitters before posting this also showed a tweet from an individual which I won’t reproduce here, but it links directly to a very extensive internal exception log on 000webhost. It’s yet another indicator of some very sloppy security practices.
Usually when I look at a data breach, I have a pretty good sense of whether it’s legitimate or not at a glance. Once you see hundreds of millions of records you start to get a knack for it! The data that was allegedly from 000webhost conformed to this tab-delimited structure:
[id] [name] [ip address] [email] [password]
It looked legit, but there’s an easy way to test and get a much higher degree of confidence in the authenticity of the data and that’s to ask if the email addresses exist on the site. I almost always find that an enumeration risk exists on the registration page. What I mean by this is that I could attempt to sign up with an email address that already exists (I always pick an obvious test one from Mailinator) and you’d see something like this:
I picked several clearly disposable email addresses randomly from the dump and got exactly the same response. The chances of this happening by coincidence are extremely low and the only other explanation that can sometimes come up is that an “attacker” has used an enumeration risk to build up a list of email addresses on the site then faked the other data (i.e. keep hitting a resource that confirms or denies an account exists and steps through a big list of emails to check). It would have been possible to emphatically confirm if the data was legit by actually trying to login with the plain text password, but that wasn’t going to happen as a matter of principle.
This was enough for me – I had to notify 000webhost so that they could advise their customers and obviously fix the underlying risk as well. And this is where it all started to get very hard…
I’m writing this blog post while speaking at events in the US (coincidentally, teaching developers how to secure their things…) so I’m going to give you the timeline in PST then express the follow-up events in days, hours and minutes after that.
12:00 midday Thursday 22: I started by trying to find WHOIS contacts but all contacts where hidden courtesy of Domains by Proxy:
I moved onto the website to look for contact info but the only channel I could identify was a “report abuse” form:
Ok, I would have preferred to email someone but let’s use the form. Here’s what happened next:
Wait – I have an account?! That can’t be right, I’m pretty sure I never created one and a quick look inside 1Password confirms that I certainly haven’t used one in recent years. Perhaps the form is just erroring out, let’s find another way to contact them, perhaps via their Twitter account. Except not much has happened there lately (or ever):
However I do find that their Facebook page is a little more up to date and it references a “premium-hosting partner”:
I also find that the footer of the 000webhost site links to them:
That jumps straight off to hosting24.com as well so let’s give them a go. I head over there and it’s a similar deal – no obvious contact info. Well that’s not entirely true, they have an image of a telephone with “24” next to it… then a fax number (they accept faxes 24 hours a day, perhaps?) plus an address in Cyprus:
But there’s also a “contact us” form.
+57 minutes after first attempting to contact them: I fill out the form:
+1 hour 15 minutes after first attempting to contact them: It only took them 18 minutes to respond which was pretty good:
Ugh, ok, so let’s go back to 000webhost. I try to submit the same message again but use the email address troyhunt+000webhost@hotmail.com which is perfectly valid and will route to my normal inbox, except…
So that’s not going to work, let’s just go and reset the password for the account using troyhunt@hotmail.com which isn’t really my account but hey, it’s my email so that’s kind of ok.
+1 hour 28 minutes after first attempting to contact them: I log a ticket directly on 000webhost under my email (which isn’t really my account):
And I waited. And waited. And never heard anything back. Ever.
+1 day, 7 hours and 12 minutes after first attempting to contact them: So it’s back to hosting24.com again and I lodge another ticket.
And 14 minutes later, they reply:
Now I’m not real comfortable with providing some unknown helpdesk person with such critical information so I try to reply and ask for a contact… except I can’t. You have to rate their reply before you can post your own reply:
No matter how much I tried, rating their reply wouldn’t give me a reply box. This is becoming really frustrating so I lodge a new ticket.
+1 day, 7 hours and 44 minutes after first attempting to contact them: I ask for escalation and contact via email:
About 17 and a half hours later, they get back to me:
System is secure enough?! I read this just as I landed in the US and I’m sitting there on the plane trying to get this really important message through to them and just not getting anywhere. I get into the airport, fire up the laptop and lodge another new ticket because I still can’t reply to existing ones!
+2 days, 4 hours and 49 minutes after first attempting to contact them: I decide it’s not worth trying to get direct and personal contact and it’s more important that they’re convinced there’s a problem. I give them enough information to verify the breach but nothing that’s too sensitive to expose to a generic helpdesk worker (besides, their system is secure enough…):
I made the reference to forwarding this to their CEO because that’s exactly what they suggest you should do:
And that was the very last contact I had with them. To date, there have been zero response from them after that last message and this is a communication channel that had previously been pretty chatty. Clearly, this is just not something they want to know about.
I spend my Sunday at a workshop in Vegas teaching a room full of developers how not to get themselves pwned. Still no feedback and I’m thinking “there are potentially 13M people having their accounts abused not just on 000webhost, but in all sorts of other places due to password reuse and these guys don’t seem to give a damn”. So I put out a tweet:
Are you a 000webhost user and have a moment to help me out with something? DM me.
— Troy Hunt (@troyhunt) October 26, 2015
I have a couple of replies and I respond with this message:
Do you mind sharing with me which email address you used? I'm trying to validate something and will share more with you if it's what I'm after.
I get some feedback but I also follow up the next morning:
Still looking for some 000webhost users to help me out with something, ping me if you have an account.
— Troy Hunt (@troyhunt) October 26, 2015
I get a bunch of replies with email addresses that are in the breach and I provide them with their data. Here are some of the responses I get:
I can indeed confirm that you have got my old IP, the correct email address and password and everything you've recovered is valid. Ouch!
Yep, that's legitimate, it's got one of my old passwords on there, which i've just confirmed.
Oh wow, that's a common one; yikes
Yeap, that's legitimate
I ask each one not to publicly socialise the information but obviously think about changing their passwords. By now there’s no remaining doubt that the breach is legitimate and that impacted users will have to know. I’d prefer that 000webhost be the ones to notify them though. And then I got some other interesting messages.
One was via someone I was having a completely unrelated conversation with:
Yep, also is it true 000webhost got compromised? Heard it from a friend and I know I have an account on there, apparently it's plaintext too so I was just wondering if you can confirm it so I can rapidly change a few of my accounts pws
Which struck me as interesting – obviously there’s some discussion going on about the incident.
Someone else contacted me with this:
000webhost was breached, original copy that you most likely have
was dumped in march
uid name ip email plaintext pw
That’s the exact structure of the data so clearly there was prior knowledge of the breach. Other people reached out as well and whilst I won’t share the details of exactly what they said purely on the grounds that private discussions deserve to stay that way, this one sentence needs airtime:
The database is selling for upwards of $2,000 right now, I can't understand which moron would be considering giving you a copy for free when people can make some serious money from this database.
I also heard from the individual who originally passed on the breach (the above-mentioned ”moron”):
I would prefer if no one notified them regarding this because friends of mine are making money from it but you're too ethical to let it go now
So consider the ramifications of this: there are potentially 13M people having their details traded for commercial purposes. The only reason anyone pays for this sort of information is because they expect an ROI; they will gain something themselves from having paid a couple of grand for the credentials. That may mean exploiting the victims’ 000webhost account but more than likely it also means exploiting their other accounts where they’ve reused credentials.
Now 4 days in since originally reaching out to 000webhost, I contact a friend who reports on these sorts of incidents. Thomas Fox-Brewster is a reporter for Forbes and he’s been great in the past at representing security incidents with balance and objectivity. I want Tom’s help in getting through to 000webhost and reporters have a knack of getting orgs to sit up and pay attention if they think a story might be written about them. Tom’s a decent guy too and I knew he’d approach the whole thing responsibly.
Tom and I talk via Skype at length and over the ensuing 24 hours he does his best to get a response. He discovers the parent company of 000webhost and hosting24.com is Hostinger which is based in the UK. That’s kind of handy for Tom being there himself so he tries to get in touch with them but they fob him off, not wanting to talk with him about the potential breach.
Tom also tries to reach out via 000webhost’s Facebook page, the one which is actually reasonably active:
Just before Tom’s message, Rob Atkinson made the post you see below his (I’ve no idea what Rob’s subsequent response to Tom is about). He was right too – as of Tuesday morning, here’s what happens when you try to login to 000webhost:
So it looks like they’ve reset everyone’s password. There’s only one good reason why an organisation does that, and that’s because they believe all the passwords have been compromised. This was the first clear acknowledgement from 000webhost that they had been breached. Of course this does nothing to protect impacted users’ other accounts where they’ve reused passwords, only communication from 000webhost alerting them to the incident will help with that.
In the hours before posting this, the Facebook comments were deleted:
000webhost invited all the Hostinger users over to their service:
I mentioned Tom contacting Hostinger earlier and them fobbing him off. Here’s a snap of their portfolio of projects:
And when you consider they’ve got the same people working across all three services, it starts to become clear how interlinked everything is:
In fact the relationships become very clear and the “free” service offered by 000webhost is put into perspective when you watch material like this:
Back to 000webhost specifically, they’ve now disabled FTP which was mentioned to Tom verbally via Hostinger and can be seen discussed on the 000webhost forum at present, including – and then confirmed – in the thread titled getting error on connection:
Until November 10?!?!
But so far, there’s still zero communication about the actual breach itself. Not from 000webhost or hosting24.com or Hostinger (and they all appear to be merely offshoots of the latter). They haven’t acknowledged me, they haven’t acknowledged Tom and now 6 days on, they haven’t even publicly acknowledged the breach other than implicitly by disabling and resetting services. They know the data is public and it’s been emphatically confirmed via multiple independent means:
- The email addresses in the breach exist on the site
- The passwords and IPs have been confirmed as legitimate by multiple account holders
- 000webhost has reset everyone’s password and disabled FTP
I probably don’t have to share exactly how I feel about how this organisation operates, it’s pretty self-evident if you’ve read through everything above. I hope this has given you some insight as to how many organisations still handle your data, how it is compromised, traded and monetised and just how hard it can be to actually get through to organisations in the wake of an incident like this.
I’ll leave you with a comment from Oliver, a fellow developer and one of the people that contacted me and verified their data from the breach:
Looking at the site, it appears like the creation of one individual or a very small team with little experience building sites at such scale; in today's day and age, security on the web simply isn't taken seriously enough.
Hard to argue with that.
There are now 13,545,468 000webhost email addresses searchable in HIBP.
Update: Also see Tom’s story about the breach on Forbes.
Update: Finally, 000webhost has notified some customers of the breach (many accounts in the data set have not received a notification):
Almost 8 days after they were first notified, here we are: https://t.co/fRvrkvabfl
— Troy Hunt (@troyhunt) October 30, 2015Update: I've been inundated by requests from people who want me to check which password they were using or who want a copy of the breach. Please read No, I cannot share data breaches with you.