In the end, I decided the fairest, most balanced way was to piss everyone off equally. Of course I’m talking about API versioning and not since the great “tabs versus spaces” debate have I seen so many strong beliefs in entirely different camps.
Imagine this:
HTTP GET: https://haveibeenpwned.com/api/breachedaccount/fooResponse:
["Adobe","Gawker"]
This was just fine. When I built Have I been pwned? (HIBP) in late November, it was intended to be a simple, fast service that a few people would use. I think it’s fair to say that the first two points were achieved, but not the last one. It wasn’t a “a few”, in fact by the end of the first week it was more than Google Analytics could handle. The point is that you can’t always predict the future when you write your API and at some point you may well need to change something that people already depend on.
But here’s the problem – every time you start talking about anything to do with APIs over HTTP, this happens:
Every which way you turn there are different philosophical takes on “the right way” and lots of backwards and forwards on REST, what is RESTful, what is not and if it even matters. Let’s talk about API changes, the impact on versioning, why there are so many diverging ideas on how it should be done and ultimately, why none of the banter is as important as actually getting stuff done.
Pulling more breach data
Keeping in mind that the same API is used for the search feature on the site and now also by external parties creating everything from smart phone apps to penetration testing tools, the response above worked fine in the beginning, but was limited. For example, this response doesn’t work quite as well:
["BattlefieldHeroes","Gawker"]
Why? Because “BattlefieldHeroes” is Pascal-cased which is great for matching it up to hard-coded CSS classes (although probably not a good long-term approach) and for having a “stable” name to refer to (I won’t change it, even if there’s a second breach), but it’s not suitable for displaying as a title. All of this comes out of Azure Table Storage and I then go into SQL Azure to pull relational data that actually describes the breach. One of the attributes in that relational storage is the name you see above.
What I really wanted to do was something more like this:
[
{
"Name":"BattlefieldHeroes",
"Title":"Battlefield Heroes",
"BreachDate":"2011-06-26",
"AddedDate":"2014-01-23T13:10Z",
"PwnCount":530270,
"Description":"In June 2011 as part of a final breached data dump, the hacker collective "LulzSec" <a href=” http://www.rockpapershotgun.com/2011/06/26/lulzsec-over-release-battlefield-heroes-data/ “>obtained and released over half a million usernames and passwords from the game Battlefield Heroes</a>. The passwords were stored as MD5 hashes with no salt and many were easily converted back to their plain text versions.",
"DataClasses":["Password","Username"]
},
{
"Name":"Gawker",
"Title":"Gawker",
"BreachDate":"2010-12-11",
"AddedDate":"2013-12-04T00:00Z",
"PwnCount":1247574,
"Description":"In December 2010, Gawker was attacked by the hacker collective "Gnosis" in retaliation for what was reported to be a feud between Gawker and 4Chan. Information about Gawkers 1.3M users was published along with the data from Gawker's other web presences including Gizmodo and Lifehacker. Due to the prevalence of password reuse, many victims of the breach <a href=\"http://www.troyhunt.com/2011/01/why-your-apps-security-design-could.html\">then had their Twitter accounts compromised to send Acai berry spam</a>.",
"DataClasses":["Email","Password","Username"]
}
]
Get it? To the previous point on the name of the breach, that’s still in there in the name attribute but now we have a title as well. This is what you show to people – “Battlefield Heroes” – but more importantly, if Gawker is pwned again I can name the breach something like Gawker2014 and the title can be something friendly along the lines of “Gawker (Syrian Electronic Army Attack)”. It segments what is stable and predictable from that which is not and it means people can create dependencies such as images or other assets on the name attribute.
The other data should be pretty clear: the date of the breach, when it was added to the system, the number of accounts pwned, the description of the breach (again, this may change if the verbiage needs to be tweaked) and “DataClasses”. One of the things many people had been asking for was a description of what was compromised in the breach so there are now a whole bunch of attributes than can be added via a collection on the breach itself. I’m already showing these beneath each breach on the Pwned websites page (this is another reason I may now tweak some of the descriptions).
This is a breaking change. Whilst the sentiment of the API is the same – provide an account name, get back a list of breaches – there is no longer a string array of breach names. If I simply replaced the old API with this one, stuff would break. APIs. Must. Evolve.
Software evolves, APIs must be versioned
Let’s just be crystal clear about this: the world moves on. The API for HIBP lasted about 2 months, not because it was badly designed but because the service became wildly, unexpectedly successful. I like these kinds of problem, and so should you.
Now I had a choice; either I could make do with what I had and deprive people of a better way, I could add to the existing service in a non-breaking fashion or I could create a new version (albeit exposing the same entity in a different way) and build it the best damn way I knew how; no unnecessary bytes, modelled correctly (until I decide a new version is more correct) and a good representation of the entity that I’m ultimately trying to get into consumers’ apps.
There is nothing wrong with introducing a new version of an API when it is the most sensible thing to do. By all means, do your damndest to get it “right” from day one but do so with the expectation that “right” is a temporary state. This is why we need to be able to version.
The various versioning camps
Right, so how hard can this versioning business be? I mean it should be a simple exercise, right? The problem is that it gets very philosophical, but rather than get bogged down in that for now, let me outline the three common schools of thought in terms of how they’re practically implemented:
- URL: You simply whack the API version into the URL, for example: https://haveibeenpwned.com/api/v2/breachedaccount/foo
- Custom request header: You use the same URL as before but add a header such as “api-version: 2”
- Accept header: You modify the accept header to specify the version, for example “Accept: application/vnd.haveibeenpwned.v2+json”
There have been many, many things written on this and I’m going to link to them at the end of the post, but here’s the abridged version:
- URLs suck because they should represent the entity: I actually kinda agree with this insofar as the entity I’m retrieving is a breached account, not a version of the breached account. Semantically, it’s not really correct but damn it’s easy to use!
- Custom request headers suck because it’s not really a semantic way of describing the resource: The HTTP spec gives us a means of requesting the nature we’d like the resource represented in by way of the accept header, why reproduce this?
- Accept headers suck because they’re harder to test: I can no longer just give someone a URL and say “Here, click this”, rather they have to carefully construct the request and configure the accept header appropriately.
The various arguments for and against each approach tend to go from “This is the ‘right’ way to do it but is less practical” through to “This is the easiest way to create something consumable which therefore makes it ‘right’”. There is much discussion about hypermedia, content negotiation, what is “REST” and all manner of other issues. Unfortunately this very often gets philosophical and loses sight of what the real goal should be: building software that works and particularly for an API, making it easily consumable.
It’s about having a stable contract, stupid!
More important than all the ranting and raving about doing it this way or that way is to give people stability. If they invest their hard-earned effort writing code to consume your API then you’d better damn well not go and break it on them further down the line.
Honestly, the debates about what is “RESTful” versus what is not as though the term itself will dictate your success is just nuts. Turn that discussion into “Here are the practical reasons why this makes sense and this is what might happen if you don’t do it”, and I’m all ears. Problem is, even the voices of reason within the noisy discussions leave doubt as to what really is the best approach and therefore I’ve reached a compromise…
Here are 3 wrong ways to consume the HIBP API you can now choose from
Ok, now that we’ve clearly established however you do it is wrong, I’d like to give you the choice to choose from any one of the 3 wrong ways. Wait – what?! It’s like this: however I implement the API, it will either be too hard to consume, too academic, too likely to fail at the proxy or too something something. Rather than choose 1 wrong way, I’ve decided to give you all 3 wrong ways and you can choose the one that is the least wrong for you.
Wrong way 1 – URL versioning:
HTTP GET: https://haveibeenpwned.com/api/v2/breachedaccount/foo
Wrong way 2 - custom request header:
HTTP GET: https://haveibeenpwned.com/api/breachedaccount/foo api-version: 2
Wrong way 3 - content type:
HTTP GET: https://haveibeenpwned.com/api/breachedaccount/foo Accept: application/vnd.haveibeenpwned.v2+json
Inevitably, someone will tell me that providing 3 wrong ways is the wrong thing to do. Won’t it mean more code kludge to maintain? No, it simply means the underlying Web API implementation is decorated with two attributes:
[VersionedRoute("api/breachedaccount/{account}", 2)] [Route("api/v2/breachedaccount/{account}")] public IEnumerable<Breach> GetV2(string account)
The first one is simply a routing constraint that implements RouteFactoryAttribute. I pass in the route and pass in the version that can map to that route then the implementation looks for the presence of either an “api-version” header or an accept header matching this pattern:
@"application\/vnd\.haveibeenpwned\.v([\d]+)\+json"
If the version specified in either of these matches the one specified in the routing constraint then that’s the method that will be invoked. This is a simple adaptation of this sample on CodePlex.
The second attribute decorating the GetV2 method above comes courtesy of Web API 2 and the attribute routing feature. Of course we could always do routing in Web API but it was usually defined globally. Attribute routing like this brings the route definition to the context where it is applied and makes it dead easy to see what controller action is going to be called by what route. It also means that the implementations for all 3 wrong ways of calling the API sit together in one neat location.
So in short, no, this doesn’t create a lot of kludge and is very easy to maintain. Each of the 3 approaches will return exactly the same result and most importantly, they’ll remain stable and won’t change in any breaking way and really, at the end of the day, that’s the most important thing regardless of which option you choose. The entire implementation is now also clearly documented on the API page of the site.
But what if you don’t specify a version?
You know the bit where I said you can’t break what’s already out there? Yeah, so that means that if you do what you do now – not specify a version – then you get what you get now. In other words, no request for a specific version means you get version 1.
I’m quite ok with that regardless of having reached this point by default. I know some people always like to return the latest version if a number isn’t specified, but IMHO that breaks the whole “stable contract” goal; what you get from the API today may be completely different to what you get tomorrow if I revise it. That would suck and it would break things.
You have 3 choices, but my personal preference is…
I have the luxury of controlling both the API and the primary consumer of it being the HIBP website itself. Given I’ve provided 3 options for consuming the API, which one do I myself use?
I’ve gone with the philosophical favourite which is to specify it via the accept header. I don’t think this is right and the others are wrong, rather I think that this makes the most sense for two primary reasons:
- I agree the URL should not change: If we concur that the URL represents the resource then unless we’re trying to represent different versions of the resource itself then no, I don’t think the URL should change. The breaches for foo are always the breaches for foo and I don’t think that just because I change the data returned for foo that the location of foo should change.
- I agree that accept headers describe how you’d like the data: This is a semantic of the HTTP spec and just as the semantics of request verbs make a lot of sense (i.e. are we getting or putting or deleting or posting), so too does the way the client would like the content represented.
By no means does this mean that I think the other two are wrong and frankly, there’s no better way to share the API with someone than to say “Here, click this”, but when I can easily construct the request and manage the headers, I’ve gone with this route.
Actually, come to think of it, I also use the version in the domain route. Why? Just through the process of writing this API I was constantly communicating with people about the ways of querying it (more on that later) and the attributes it returns. Being able to go flick an email and say “Hey, here’s what I’m thinking” and they simply click it and get results is invaluable. This is the point that proponents of the URL versioning approach quite rightly make: you simply cannot do this when you’re dependent on headers.
Oh, and in case you’re checking up on me, at the time of writing I haven’t yet rolled the site over to v2 of the API. Now that the breach data is pulled back in the API when a search occurs it means I have the luxury of not loading all breaches into the source on initial load (that was never going to be sustainable as the data set expands). This’ll save a bunch of outbound traffic and speed things up for people in terms of getting the site loaded, but it also means just a little more work on my end. Stay tuned.
In closing
Clearly I’ve been a little tongue-in-cheek here with regards to everything being wrong but honestly, the more you read on this and the more questions you ask, the more wrong every route seems in one way or another. In fact I know damn well that there are aspects of my implementation that will be referred to as “wrong” (I can think of at least a couple) and naturally I’m bracing for the potential onslaught of feedback to that effect. The thing is though, each of these options works and frankly, for all practical purposes, they work just as well as each other.
If I may leave others considering how to version their APIs with a final thought: Nobody will use your API until you’ve built it. Stop procrastinating. None of these are “bad” in any tangible sense, they’re just different. They are all easily consumable, they all return the same result and none of them are likely to have any real impact on the success of your project.
References
- Stack Overflow: Best practices for API versioning? (great question, great answers, closed as “not constructive”, I assume because “Bill the Lizard” got out on the wrong side of bed that morning)
- Lexical Scope blog: How are REST APIs versioned? (good comparison of versioning practices across services. albeit now a couple of years old)
- CodePlex: Routing constraint sample (linked in from Microsoft’s Web API page as an example of versioning APIs by adding a custom header)
- CodeBetter.com: Versioning RESTful Services (very pragmatic and a good description of the various ways an API might change)
- Vinay Sahni’s blog: Best Practices for Designing a Pragmatic RESTful API (he’s arguing for URL versioning for the sake of “browser explorability”)
- Pivotal Lans: API versioning (good view of the conflicting opinions out there)
- Web Stack of Love: ASP.NET Web API Versioning with Media Types (good end-to-end walkthrough of creating an app to support versioning by content negotiation)