This weekend, I loaded five additional data breaches into Have I been pwned (HIBP) that had come from various forums running on vBulletin. These came via supporters that had collected them from data breach traders over the years and some of them dated back quite some time. I always go to great lengths to validate that a breach is indeed legitimate and one of the ways I do that is to take a real good look at the passwords stored in the system and ensure that they do indeed adhere to the sorts of password patterns we’re used to seeing (i.e. poorly chosen and often including the name of the site). Fortunately for my purposes here – and unfortunately for those who actually had accounts on these sites – vBulletin does a pretty sloppy job of storing passwords and I thought I’d use this as an opportunity to demonstrate just what I mean by that. I’ll also show how “salted hashes” can be created in a very weak fashion as used by vBulletin or a very strong fashion using modern adaptive algorithms.
Let’s take the Gamerzplanet breach which had over 1.2M accounts in it with the first ten passwords appearing as follows:
af48332ec7bae3b43c2f8c28f1b6479e:f\u
befd08dfd48fc13a47f5dcd467f4964f:$:}cL1SDYX$s+viF%MJ{w(W|nmG%IS
a92f7415b68d649f0e0314b149a8bc0a:5ze
10a9ef2ec83234eec337e557333f78f5:/Do
2fc88d6b7b827bec242a307604c1c161:27L
0101a55a81ff2a185d6a758ab0bce632:4fWN2/+I~&AW]UrMw\AM3pZJR55b#O
cb574910766230d9e4bfa979110b26e2:~KJ
504661de0decb40df6feaac4eed46884:|p|
618fbea1123e82ff547274e3134c0731:0 c
5e155f2a3b61528bd2c772ce7230ce35:@9]
The first thing you’ll notice here is that two of the rows are much longer than the others. What we’re actually seeing here is fixed length MD5 hashes on each row then a colon delimiter between the hash and the salt. Two rows have salts of thirty bytes, the other eight are only three byte long salts. Without knowing exactly what framework these had come from (although having my suspicions based on it’s prevalence in HIBP), I plugged the first value into a hash identification service:
It didn’t take long to work out that these were indeed vBulletin hashes, the shorter ones being pre v3.8.5 and the longer ones being post (obviously they’d upgraded the forum at some stage and changed passwords were hashed using the latter approach). This meant I could then dump the hashes out from the source file (I use Log Parser for this sort of activity) and proceed to crack them which would verify that they did indeed originate from vBulletin. I grabbed hashcat (which is now open source, by the way) and then checked out the example hashes page and found vBulletin with the hash modes 2611 and 2711 for the older and newer versions respectively. I kicked off with the earlier gen ones:
This approach to cracking involves the GPU in my AMD R9 290X (now getting on a couple of years old) calculating alphanumeric hashes between 6 and 10 characters long and comparing them to the ones in the breach. You can see from the speed above that we’re working through them at 3,012.4 MH/s or in other words, calculating more than 3 billion (yes, with a “B”) hashes per second. Crikey.
What you’re seeing above both before and after the stats is the hash, the salt and then the plain text password to the right. The first password in the console above is “soccer”, the next one “killer” and so on and so forth. As I sit here watching the progress, the plain text versions of these passwords are just flying across the screen. Remember, these are salted hashes, not just direct hashes of the password with no uniqueness added which is what the salt brings to the table. Those earlier vBulletin passwords have two MD5 hashes each according to the hash identification service above too. Do also note the complexity of some of those hashes – 123456, for example – which goes to show that not only do people continue to choose absolutely lousy passwords, but systems still allow them to!
Or course doing this puts quite a load on the GPU with the utilisation immediately going to 100% and the temp rising accordingly:
Mine is water cooled so runs silently and maxes out at about 75 degrees C which is just fine. It could sit there and do this all day long, day in and day out, gradually working through all the possible hash values. It’ll never crack all of them, but it doesn’t have to – how many compromised passwords are enough to deem the approach to storage “insufficient”? Half of them? A quarter? A tenth? There’s no exact science to this but the expectation should always be that the hashing algorithm is resilient to this style of attack such that only a very small number are crackable within a “reasonable” time and even then, only very poorly chosen passwords should be falling victim.
I let the process run for about 22 hours and as it progresses, the rate of cracking slows but I still have a sizeable whack of cracked passwords:
During this time it calculated 52,175,799,582,720 hashes, that’s just over 52 trillion in case all the commas make you lose track. But it’s just getting started; next I gave it a password dictionary, namely the hashkiller.dic one with over 26 million passwords in it. These are just plain text strings that people have used as passwords in other systems and the output looks something like this:
There’s more variety in a dictionary of this nature and because it’s put together with passwords that people typically use (often these are constructed from breaches where there was no cryptographic storage), the hit rate per hash is much higher then the auto-generated stuff. On the flip side, the rate here is much slower at “only” 586 million hashes per second as the GPU now needs to be fed in values to hash rather then just working through a character range as it did earlier. Hashcat also knows which passwords have been cracked already and the results are simply appended to that. Once it finishes, we’re at a total of over 60% cracked:
You can go on and on with various other types of attack and keep chipping away at the remaining hashes. There’s a thread over on the hashcat forums which talks about a bunch of different approaches and it’ll give you an idea about how the process works. (Incidentally, it also demonstrates that I should have done the dictionary attack first given it’s much higher success rate and shorter duration.) For now though, this pretty much makes the point – over 135k password hashes cracked in less than a day by me casually throwing hashcat at it. Put it in the hands of someone who’ lives and breathes hash cracking and you’ll see a significantly higher number. Give to someone who really knows what they’re doing – someone such as Jeremi Gosney from Sagitta – and he’ll throw stuff like this at it:
Now that’s not to say that a mere set of hashes from a gaming forum warrants this kind of attention, rather it’s to illustrate that what I’ve shown above is merely the entry point for this sort of thing and it still tore apart the majority of vBulletin’s hashes in next to no time. In case you’re wondering, the newer implementation of their password hashing I mentioned earlier resulted in “only” about 2 billion hash calculations per second so in real world terms, it’s only slightly better than the older format and well within the spectrum of what I would call “almost useless”.
Last thing on vBulletin – there are heaps of exploits against various versions readily discoverable after a few minutes of Googling. I very quickly found walkthroughs that leveraged union based SQL injection:
Which in an example like this, resulted in the hashed admin password and salt being returned from the database:
Anecdotally, I get the sense that many of the breaches I’m seeing against these systems are simply due to attackers jumping from vBulletin forum to vBulletin forum and trying their luck with known exploits. Evidently, many of these attempts prove successful and it’s frankly alarming that there are some many vulnerable installations out there. I doubt very much that many are actually maintained too; certainly vBulletin issue patches when bugs are identified, but how many of these installations are actually kept up to date? Even vBulletin themselves were hacked a few months ago so what chance do others running the forum software have at protecting themselves?
Let me round out on a more positive note though and it’s about where we’d like to see password storage moving to. OWASP has a great resource titled the Password Storage Cheat Sheet and they talk extensively about “work factor” or in other words, making it slower to calculate hashes. Implementations such as PBKDF2, scrypt and bcrypt make the world of difference to the resiliency of compromised hashes. After the Ashley Madison breach last year, various people took shots at cracking the bcrypt hashes (this was before discovering that they didn’t even need to). The sorts of speeds people saw in that exercise were more in the order of a few dozen hash calculations per second. Think about that for a moment – those vBulletin hashes were being calculated somewhere in the order of 77 million times faster than those stored with an appropriate hashing algorithm. That puts an absolutely ginormous dent in the ability to do anything useful with compromised credentials and whilst it won’t bring the risk down to zero, it gets us way closer to it.