If I'm honest, I'm constantly surprised by the extent of how far Have I Been Pwned (HIBP) is reaching these days. This is a little project I started whilst killing time in a hotel room in late 2013 after thinking "I wonder if people actually know where their data has been exposed?" I built it in part to help people answer that question and in part because my inner geek wanted to build an interesting project on Microsoft's Azure. I ran it on a coffee budget (the goal was to keep the operating costs under what a couple of cups from a cafe each day would cost) and I made it freely accessible. And then it took off.
As this service has grown, it's become an endless source of material from which I've drawn upon for conference talks, training and indeed many of my blog posts. I've written extensively about how HIBP has grown over the years and doing so has been a cornerstone of the philosophy of how I've run the service - with maximum transparency. My view has always been that it's in everyone's best interests to be crystal clear about how I run this, especially when you consider the circumstances of how most of this data was leaked in the first place. And this is precisely why I'm writing this piece - to talk about how I'm assisting the UK and Australian governments with access to data about their own domains.
Just to scroll back for a bit of context, anyone who owns a domain can do a free domain search on HIBP. There's a verification process where control of the domain needs to be demonstrated (email to a WHOIS address, DNS entry or a file or meta tag on the site), after which all aliases on the domain and the breaches they've appeared in is returned. At the time of writing, over 110k domain searches have been performed and verified. These searches span every imaginable class of domain; financial institutions, aerospace, healthcare, adult entertainment and based on a very rough check just now, more than a quarter of all Fortune 500 companies as well. Amongst those verified domain searches are government departments and they too are enormously varied; local councils, legal and health services, telecoms and infrastructure etc. The thing is, loads of government departments within different countries have all been running these searches independently and that means an awful lot of duplication of effort has been going on. This post talks about how I'm addressing that.
Over recent times, I've had a bunch of opportunities to talk to folks in various government roles. My congressional testimony in the US was a very public example of that, less so are the dozens of conversations I've had in all sorts of settings including during conferences, workshops and over coffees and beers. The subject outlined above (loads of government departments independently using HIBP) came up in a number of those meetings, so we decided to do something about it. Not only did we want to consolidate all those existing independent departments doing their own thing, we wanted to expand the scope to all government departments. So, this is what we've done:
As of now, all UK government domains are enabled for centralised monitoring by the National Cyber Security Centre (NCSC) and all Australian government domains by the Australian Cyber Security Centre (ACSC).
The way we're doing this is by using the commercial model within HIBP and scoping it to a limited set of whitelisted domains. Whilst I've written and spoken publicly about commercial services in the past, I've avoided promoting them per se which is why you won't find anything on the HIBP website or any up-sells in emails or anything along those lines. I've always wanted HIBP to be first and foremost a freely available service for email and verified domain searches and particularly in this industry, it's very easy for financial motives to taint the ethics of how this data is dealt with. To that point, I've made this available to the NCSC and the ACSC without any commercialisation whatsoever - they get it for free. There are many reasons why that made sense to do, one of which is that it unifies a bunch of existing free searches that I mentioned above. Another is that frankly, we really want governments to do their best to protect the folks working in their departments; many of them are working in capacities that help protect our respective nations from all sorts of threats and increasingly, as we all know, that means online threats as well.
Getting back to the mechanics of things, the respective govs are using the commercial HIBP model in a tightly scoped fashion. For example, the UK government can query any .gov.uk domain on demand and the Aus government can query any .gov.au domain on demand. They can both also query a small handful of whitelisted domains on different TLDs, for example, The Commonwealth Scientific and Industrial Research Organisation (CSIRO) runs on csiro.au so that domain is whitelisted for the ACSC in addition to the .gov.au TLD. What this means - and this is enormously important - is that the NCSC and ACSC can't turn around and query, say, troyhunt.com. The only access they have is to domains that their people working in those departments could query anyway via the existing free domain search model, we're just consolidating it all into a unified service.
As part of that service, they'll also be using the existing notification service that commercial subscribers have access to. This is a webhook model which calls back into an endpoint the respective governments host. Every time an alias on one of their domains is seen in a new data breach or a paste, the incident is automatically posted to them. It means that within minutes of one of their email addresses being found and loaded into HIBP, they'll know about it. That's really important in terms of giving them the ability to respond quickly and by unifying all those existing one-off domain searches, the respective governments will be able to immediately see when an incident has a potentially broad impact. This can be especially important when you consider data breaches such as Dropbox; many organisations of all kinds suddenly learned that a bunch of their people had cloud storage accounts under their corporate email addresses so you can imagine some of the discussions that subsequently ensued.
So that's what's been set up in HIBP for the UK and Aussie governments. They both respected my desire for transparency and understood why it was important for me to write about it publicly. I'm happy that this effort continues the philosophy I've stuck to since the early days of HIBP - that the service should help people do good things after bad incidents occur and that it does so as transparently as possible
Lastly, I want to touch on something a bit tangential and that's to point out some of the great work these agencies are doing to try and improve online life for the rest of us. I've regularly quoted the NCSC in particular, for example there's a bunch of their work in my recent blog post about authentication guidance for the modern era. I love that we have a government department making recommendations such as "only ask users to change their passwords on indication of suspicion of compromise" because it validates what an increasingly large number of us in the security industry have been saying for so long. The NCSC piece on let them paste passwords is another favourite; the blog post of mine they reference there has led to a regular cadence of people pointing out sites that don't adhere to this guidance and having a government resource for me to point offending companies at is enormously valuable. Likewise, in Australia we have Stay Smart Online which provides a bunch of consumer-level information about precisely what the name of the site suggests. They regularly highlight emerging scams and other digital threats to everyday Aussies as well as creating practical guidance for the fundamentals such as guidelines for creating strong passwords (I particularly like the fact this draws on the NIST recommendations I included in the aforementioned authentication guidance blog post). I'm very happy that HIBP is now a resource the UK and Aus governments can draw on to help their people help all of us live happier (and hopefully less pwned!) online lives.