Mastodon

On The (Perceived) Value of EV Certs, Commercial CAs, Phishing and Let's Encrypt

19 July 2017

Last week I wrote about how Life Is About to Get a Whole Lot Harder for Websites Without HTTPS. Somewhere in the comments there, the discussion went off on a tangent about commercial CAs, the threat Let's Encrypt poses to them and subsequently, the value (or lack thereof) posed by extended validation (EV) certificates. That discussion boiled over onto Twitter with many vocal opinions from different camps. This post attempts to lay the arguments out in a more cohesive fashion than Twitter permits.

But firstly, let's get back to the original blog post which I made due to the fact that come October, Chrome 62 will begin doing this:

Chrome 62 showing more "Not secure" warnings

There are two important things happening here:

  1. Any page including a form element loaded over a non-secure connection will display "Not secure" (at least once you edit the field)
  • Any non-secure page at all loaded in Incognito mode will also display "Not secure" (and it'll do it all the time)

This is good for web security as it incentivises site owners to properly protect their transport layer with HTTPS. Some people actually argue that the browser vendors should be pushing even harder on this front, flagging all HTTP pages as potentially dangerous regardless of Incognito mode. The thing is though, doing this too early dilutes the value of the warning because if people see it too often then they'll simply start ignoring it. It's all about timing.

The timing of Chrome's change comes in an era where as I proposed in January, HTTPS Adoption Has Reached the Tipping Point. I outlined many reasons for this including Chrome 56 beginning to flag any HTTP pages with a logon or payment form as "Not secure", ongoing abuses of non-HTTPS traffic and perhaps most significantly, the rise and rise of Let's Encrypt:

Let's Encrypt Growth

As impressive as that graph is, a figure not represented here is that just a few weeks ago, Let's Encrypt issued their 100 millionth certificate. It's easy to forget that Let's Encrypt only began issuing certs to the public at the end of 2015 and only moved out of beta in March last year. However, it wasn't until the beginning of this year that the numbers really start to rise dramatically; inevitably they needed to build trust and, of course, in many cases a Let's Encrypt cert wouldn't be requested until people actually needed to renew an existing one.

The meteoric rise of Let's Encrypt is undoubtedly due to their three stated objectives:

Let’s Encrypt is a free, automated, and open Certificate Authority

Free is good because it removes the barrier to entry that stopped so many smaller sites from adopting HTTPS in the first place. Automated is good because it can be painful to manually obtain and configure a cert (and you generally have to do it again every year too). And open is good because it makes the implementation easily consumable by any provider who wants to offer free and automated certs. So everybody should be happy, right? Well...

Remember what we used to do when we needed a cert? We'd head off to the likes of Comodo and pay them money, often around the US$70 per year mark. But now we have this upstart which has gone and issued a hundred million of them for free! Some people would love to say this is $7B worth of certs at "normal" market rates, but of course the adoption is only as high as it is due to the very fact that the commercial barrier has been removed (plus the certs are only valid for 3 months so the renewal cycle is less than many commercial options). Still, it stands to reason that they've taken a significant chunk of cash out of the commercial CA market. But has it really hurt them that much?

As I mentioned earlier, Let's Encrypt's greatest growth period has been this year so let's have a look at what's happened to Comodo over that period (they're still the world's largest CA at the time of writing). Here's the market share trend for the year so far:

Falling Commodo Market Share

For 7 months in a row now, Commodo's share has been falling. Being the largest they're a good place to start, but it's the same story as we go down the order, for example with Symantec:

Symantec Market Share

We could keep going - GoDaddy is next:

GoDaddy Market Share

The "best" months the 3 largest commercial CAs have had this year has have been months where market share was flat rather than declining, and only GoDaddy managed to achieve that. Now in fairness, these are all declining shares of an increasingly large pie (HTTPS adoption of the top million websites more than doubled in the 12 months to Jan 2017), but it means that a very large chunk of that pie is now being gobbled up by IdenTrust:

IdenTrust Market Share

This is simply because IdenTrust has cross-signed Let's Encrypt's intermediate certificates so their growth is tied to that of Let's Encrypt.

All of this has occurred in an era when Let's Encrypt hasn't offered wildcard certs and certainly hasn't done EV certs. That's about to (partly) change though because in Jan, Let's Encrypt will add wildcards to their offering and you can only imagine that will accelerate the growth of free certificates and conversely, the decline of commercial ones. It's pretty obvious who's going to be unhappy about this and oh boy, are they pissed.

In June last year when the penny was obviously finally dropping that free certs were gaining traction, news emerged that Comodo attempted to trademark the names "Let's Encrypt," "Let's Encrypt with Comodo," and "Comodo Let's Encrypt". Actually, Comodo's attempt to trademark the names went all the way back to October 2015 so obviously they'd seen the threat of Let's Encrypt coming very early. So how did Comodo justify what was very clearly a grab for a name that had already been established by the non-profit? Well, their CEO described the legitimacy of their claim very succinctly:

How can you prove it was them who made [the brand name] up?

That was in a post on Comodo's own forum titled "Shame on you, Comodo!" and he went on to make some pretty outrageous claims. For example, that Let's Encrypt was copying their business model because the certs expire after 90 days (he didn't comment on which CA originally "invented" the 1-year validity period). There was immediate backlash once their trademark attempt was discovered with the press calling it Super Slimy and many people promising to ditch their Comodo certs. Eventually, Comodo withdrew the trademark registration but obviously when you see behaviour like this, you start to seriously question integrity (which incidentally, is a pretty important attribute in infosec).

I saw this anti-Let's Encrypt pattern reflected in comments on last week's post, most notably by an individual representing LeaderSSL who explained in his initial comment that he was "a proven reseller with more than 10 years of history". Keeping in mind the threat that Let's Encrypt poses to that model, his claims included:

  1. Free does not mean better quality (he's right in the same way that "commercial does not mean better quality" when it comes to certs)
  • Let's Encrypt has known phishing problems (he's also right in the same way as you'll see that Comodo has in a moment)
  • Let's Encrypt requires scripts to execute on the server which is not transparent (this is false and "open" means exactly what it sounds like - complete transparency)
  • Let's Encrypt is not suitable for commercial sites (he doesn't substantiate why, plus neglects to mention sites like Theme Forest, the online marketplace protected by Let's Encrypt and the 444th largest website on the internet)
  • A "crone breakdown" [sic] would put your business in danger (automated renewal failure would break things, just like manual renewal failure would... except less likely!)

There are other nonsensical arguments in there as well which I'll save you from here but as with the Comodo situation, clearly Let's Encrypt is threatening their business model.

A cornerstone of the anti-Let's Encrypt movement has been the use of their certificates for malicious purposes, particularly in phishing campaigns. Certs are valuable for building trust in victims because as you'll see at the top of your browser right now, the presence of HTTPS provides increased visual assurance of the safety of the site. You'll see the word "Secure" next to the address bar in Chrome on the desktop, a padlock next to the URL in Safari on iOS and other similar implementations in different clients. The thing is though, Let's Encrypt are far from having a monopoly on certs issued for malicious use:

Blocked Phishing Certificates

Now this is no more Comodo's fault than it is Let's Encrypt's, rather it's a reflection of what happens when you make certs freely available. You already knew that Let's Encrypt does that, but you may not have known that Comodo also does:

Free Comodo Certificates

It doesn't tick the "automated" and "open" boxes like Let's Encrypt does though and just like the rest of us, nasty online criminals like convenience! (Incidentally, read Eric Lawrence's Certified Malice blog post to better understand the challenges we face in dealing with phishing sites using HTTPS.)

One really important part of the narrative here is that a CA issuing certs which are then used for malicious purposes in no way jeopardises the viability of that CA for other customers. Instead, opponents of Let's Encrypt focus on what they believe is a social responsibility, namely them policing site contents and revoking certs, an action which strictly speaking is not part of their remit. (Comodo frequently references their decision to revoke certs by policing phishing sites.)

But regardless of the CA, the ready availability of these certs that can be used for malicious purposes clearly poses a challenge due to user-conditioning to recognise the padlock icon as a sign of trustworthiness. The thing is though, this is not what it means nor is it ever what it has meant:

Scott is spot on here and it's at this point that we need to separate two fundamentally important yet different concepts:

  1. The use of HTTPS to protect traffic in transit
  • The use of HTTPS to provide identity assurance

Regardless of how many certs any CA issues to malicious parties, you cannot change the fact that in doing so it protects more traffic more of the time. This is the uncomfortable truth with encryption in general and certainly it's one that governments around the world are struggling with at present (most of my Friday disappeared talking to the press about this after our government made a lot of noise about the subject).

As it relates to phishing sites, another uncomfortable truth is that the CAs are right to issue them DV certs; if the site meets the requirements for domain verification then there is no expectation set forth to either verify the legitimacy of the site's business model nor who the owner of it is. That applies to both issuing the certs in the first place and revoking them insofar as it is not the CA's role to pass moral or legal judgement on the site. These are "domain validation" (DV) certs, not "domain validation and also make sure they're good guys" certs!

Further to that, CAs refusing to issue certs to domains containing words like "paypal" is a slippery and futile slope because there are so many easy variations. Do you block "paidpal" as well? And "paypaI"? You'd have to manually review them and with Let's Encrypt issuing 100k certificates a day, that's not only nigh on impossible but would be extremely financially burdensome so there go your free certs, all because of a few phishing domains using HTTPS to do precisely what it's designed to do! And then even if you did, with that free wildcard support coming to Let's Encrypt it's trivial to serve certs over Paypal subdomains. The only saving grace on all this is Certificate Transparency (CT) so at least you know which certs have been issued for funky looking names. But because we're talking DV, you still don't know who they've been issued to...

Which brings us to identity assurance and to Scott's point about you potentially having a (private) conversation with Satan. Whilst DV certs give us assurance that we're communicating with the domain we think we are, it's EV certs which give us confidence we're communicating with the organisation we think we are. For example, when you go to Have I been pwned (HIBP), you can be pretty confident that this is actually my site and not one of many dodgy rip-offs:

HIBP with an EV

I wrote about my journey to an EV cert back in December. I spent a great deal of time going through the process of requesting the cert (it was particularly difficult without an existing registered legal entity) and then a lot of time since then thinking about the whole thing. Frankly, the usefulness of an EV is highly debatable and there are a number of very good reasons for this.

The first problem is that EV certs are a human control, that is they require people to make judgement decisions based on their presence. Actually, it's sort of the inverse in that they require people to make judgement decisions based on their absence - "Hang on a moment, this website isn't showing the identity of the owner, this isn't right!" The problem is, humans are terrible at doing this:

What this means is that if instead of seeing the earlier image of HIBP someone instead saw the following, they would almost certainly trust it just as much:

HIBP with a DV

Now there is a whole other debate here about whether or not the browser should say "Secure" at all because it can be enormously misleading. For example:

Melih Abdulhayoglu is Comodo's CEO (you may remember him from earlier), and in this instance, I agree with him. The presence of the word "Secure" and the green padlock create a greater sense of trust in the site because that's what people have been conditioned to believe. There's a very worthy debate to be had here (albeit a long-term one) about how browsers should behave in a "secure by default" future. It's a debate about what role browser manufacturers should play in defining DV (the connection is secure) versus EV (the connection is secure and you know who you're talking to). As worthy as that discussion is, it makes absolutely no difference to the status quo today so I'm not going to get bogged down in discussions about that.

There are other points I certainly see eye to eye with Melih on as well. For example, earlier in the week he pointed to a perfect example of a phishing site exploiting the trust users have in the padlock icon:

But this also demonstrated a couple of other things, the first being that per my earlier comments, simply blocking names that could be used in a phishing attack is vastly insufficient. In this case, the name of the brand is actually spelled "netfilx" which is a minor deviation but it's enough to circumvent simple pattern matching and still be effective as a recognisable, trustworthy name. Then there was the certificate that was actually used:

Netflix Phishing Site Using Comodo Certificate

This makes a very salient point about the use of certificates for malicious purposes, although perhaps it wasn't the best one for the CEO of Comodo to have picked:

The very next day, the DNS for the site was pulled just like it would be for a phishing site served over HTTP. But until that time, just as with a Let's Encrypt cert, Comodo's cert helped enable malicious actors to steal credentials from unsuspecting victims. But also, as with Let's Encrypt, I in no way blame Comodo for this, they simply issued a DV cert to someone who proved they control the domain.

Back on the premise of users not behaving differently in the absence of an EV cert, I really wanted to try and put some numbers around this so I put out a little poll earlier this week:

Of course, this should come as no surprise but I wanted something - anything - to try and quantify the actual usefulness of EV because a large part of the problem is that it's enormously hard to measure. But there were two things that did surprise me about this poll:

  1. The number of responses indicating a lack of awareness of any cert
  • The replies to that tweet indicating that even tech people weren't familiar with EV certs:

What we're seeing here mirrors one of the few proper studies done on EV certs: An Evaluation of Extended Validation and Picture-in-Picture Phishing Attacks. More than a decade ago, they concluded what should already be clearly apparent by now and over the passage of time, nothing really seems to have changed:

Unfortunately, participants who received no training in browser security features did not notice the extended validation indicator and did not outperform the control group.

The challenge here is that because the value of EV lies in visual recognition, people need to be trained to expect it and then to change their behaviour if it's not present. Again, we can argue all day long about how things should be, but this is today's reality. We're simply not conditioning people to look for EV certs and here's a great example of that:

And in case you've seen this and then gone and checked each site for yourself and thought "Aha - Twitter does use an EV", that depends on where you are in the world:

But you still checked just to be sure, didn't you? Based on the responses to my tweet with the top 10 sites, many people did check and I've gotta be honest, I had no idea myself until I loaded each one into the browser. Twitter personifies the very crux of the issue with noticing the absence of an EV cert: I travel around the world extensively and I use Twitter every single day many, many times over and until now, I never knew that the EV cert is region specific. I would have seen that same site thousands of times both with and without the identity verification in the browser and even for someone who thinks about security day in and day out, I never noticed this behaviour. How on earth can we expect your average person to?!

Getting back to those top 10 sites, I was genuinely curious as to why the big players shunned EV certs but so far, nobody seems to know including Melih himself. If you do have a balanced and impartial answer to this (especially if you're from one of those 10 companies!) do please leave a comment below and feel free to make it anonymous if you'd prefer.

One very tangential argument I've seen as to why the big players mentioned above aren't using EV is because their brands are already well-established enough to not require the additional identity assurance it brings. The problem is, that even those ardently pushing for EV certs don't appear to recognise the flaw in their own logic, for example this response when asked why an EV would be required:

In isolation, this argument for the value of EV sounds fine, at least until you actually look at Amazon and realise that as in the earlier image of the top 10 sites, there's no EV:

Amazon With No EV

"Ah", but the EV advocates will say, "People already trust Amazon and what you really need an EV for is those smaller and mid-tier sites who are trying to compete with the big guys". The problem is that this position falls down at many levels and the first is that it totally neglects that age-old argument about EV protecting against phishing. Amazon is obviously a valuable target for phishing attacks and here they are, serving content with a DV. It also falls down because as we've already established, people simply aren't looking for EV certs as a sign of identity assurance. And finally, it simply doesn't align with the reality of the size of the sites that actually use EVs:

What we're seeing here is that EVs are most frequently used by larger sites and as size declines, so does EV adoption. Now perhaps the commercial CAs are simply seeing this as a large addressable market for their product (which would bring us back to financial motives again), but clearly their view of who actually needs the cert the most is not consistent with those who are actually buying them.

Moving on, the financial sector has traditionally embraced certs of all kinds earlier than most other verticals. As it relates to EV certs though, it's a bit of a mixed bag and I'll start here:

We're looking at the home pages here so this is the first thing you see when going to the respective company's websites. Now frankly, these pages should all be served over HTTPS regardless of what type of cert they're using, but the absence of HTTPS in general and EVs in particular shows that clearly, identity assurance isn't a particularly big issue to any of them at present. Once you drill down to the login pages (assumedly after clicking a link to it on the insecure page...) you are indeed presented with an EV on each one. There's certainly a higher presence of this class of cert in the financial sector but even then, adoption is spotty:

Now here's an interesting little challenge for yourself: without looking, do you know which services you use on a regular basis - financial or otherwise - serve an EV cert? Hand on heart, I have to say that I honestly don't know myself and as I said earlier, I think about this a lot more than most people too. What it means is that when we go to say, discover.com and we're presented with a DV cert as Stephen has observed, do we stop and say "Hang on a moment, I'm only seeing a padlock and the text "Secure", this isn't right!"?

Discover With no EV

No, just like with Twitter we don't and therein lies the problem. It doesn't matter that there are many financial sites out there that do use EV certs because we're conditioned to trusting DV just as much anyway.

Yet another problem is that once you do see an identity by virtue of it appearing next to the address bar, is it one that actually builds confidence or potentially even erodes it? For example, when I originally set out on the aforementioned journey to an EV cert, my initial efforts involved registering a business name under my proprietary limited trading company name which would have resulted in this appearing in the address bar:

EV Superlative address bar

This would have been very misleading and I made the following comment at the time:

I didn't want the company name in there, not because I wanted it hidden for any nefarious purpose, but because it just didn't make any sense. The whole point of the EV cert is to build trust by providing a name that people recognise and can look at and say "ah, I know who that is" and this just wasn't going to work.

I was reminded of this over the weekend after the Twitter account for the blog on The SSL Store took issue with my pointing out the lack of EV on the homepage of those top 5 banks. (I said there was no EV on the landing pages because they don't even serve over HTTPS, they said there was EV on the login pages, bottom line is we were both right.) The SSL Store is another example of an organisation under threat from free certificates as their business model is centred around selling certs. Clearly, Let's Encrypt is disrupting this industry and (somewhat understandably), The SSL Store is a little cranky about this. This is the same anti-Let's Encrypt argument as I outlined earlier, namely that they've issued certs which were subsequently used in phishing attacks. (Curiously, they don't mention how many certs Comodo has issued to phishing sites like the one Melih drew attention to earlier on, but you'll find many of their products listed for sale on the site.) The by-line of the story talks about "14k SSL certs issued to PayPal phishing sites" which sounds quite scary, but less scary is the alternate headline of "0.014% of Let's Encrypt's certs were issued to PayPal phishing sites". Even less scary again is that phishing sites served over HTTPS are actively combated in many of the same ways as phishing sites served over HTTP, just like we saw with the Comodo example. I don't know how many of those 14k sites using Let's Encrypt certs have already had their hosting pulled, been blocked at the ISP level or flagged as malicious in the browser and unfortunately, the article didn't elaborate. Regardless, that a CA - any CA - is issuing domain validated certs to sites that successfully validate their domain is not something that's going to be changing any time soon.

Getting back to the EV discussion, here's The SSL Store's site:

The SSL Store With Another Identity

Now frankly, this confused me because here I was going to "The SSL Store" but by virtue of the EV cert, my browser was telling me that no, I wasn't on The SSL Store rather I was looking at "Rapid Web Services". I have absolutely no idea who these guys are: there's no brand recognition and I certainly don't have any additional trust in the service by seeing a name I wasn't expecting. You could argue that they're more likely to be legit because EV poses additional hurdles that a non-legit site is unlikely to be able to meet, but you can see where this whole thing starts to create confusion.

Compare the approach above with that of Comodo:

Comodo With an EV

Ignoring the "Creating Trust Online" for a moment (turns out it's just eye candy), you do certainly know who you're talking to here. But we need to remember that this amounts to nothing more than a visual indicator in the client and that means it's up to the browser to decide how it's represented. In some cases, that may mean no representation at all:

Comodo EV on Chrome in Safari

This is Chrome on iOS and as you can see, the identity of the site owner no longer appears. What this means is that browsing Comodo in this fashion gives you the same level of assurance as browsing the phishing site their CEO tweeted about earlier on:

Phishing Site with Cert

Because the representation of an EV is purely a browser control, you're going to see very different results depending on the browser you've chosen:

That's if you've chosen a browser at all - these days a huge proportion of communication over the web is done via rich client apps which make no assurance of transport layer encryption at all, let alone of any sort of identity assurance:

Now we could all have a very robust discussion about what visual indicators Chrome on iOS should show but per my earlier comments on it being a much longer-term discussion, this is the present-day reality we're faced with and this is the experience that people using our websites have. I feel that's where a lot of these discussions the last few days have gone off the rails with people saying "Well, browsers should do this and padlocks should be changed to that and the secure word could maybe be replaced by..." and that's great if you want to get involved in those discussions, but it doesn't change the reality we're working with today. Most of my time is spent with people trying to deliver solutions that work now which is why I felt it was so important to capture the present situation in detail.

As I was researching for this piece, there was one particular pattern I consistently saw over and over again. On the one hand, you have proponents of EVs constantly beating the drum about increased assurance:

Consistently, they're the parties with a financial interest in the success of the more expensive certs. They're also consistently critical of Let's Encrypt making certs freely available and not policing their use; the tweet above references the earlier piece I mentioned from The SSL Store and you just start to feel like there's a cabal of parties with a common commercial interest feeling threatened by the little guy.

But part of that pattern was also the consistent message from expert independent parties about how little value EV really has. For example, Adam Caudill wrote Looking for value in EV Certificates and made precisely the same observation as many others have:

The problem with this is, users generally don’t know what that green bar is, or what it means. If it disappeared, would they even notice?

Or there's the post from Barry of Tune the Web on what the green padlock really means who said the following:

There is little definitive evidence that EV certificates provide any value to websites. While some talk about an increase in user trust and conversion rate, few studies are available for this and those that are, are usually published by those with vested interests (CAs) and are disputed.

Time and time again, the same pattern emerged and honestly, that makes a reasonable person start to question the motives of commercial entities being at odds with independent parties. Likewise, time and again, those partaking in the discussion who questioned the value of EV certs attempted to find quantifiable evidence of their value but to date, none has been forthcoming:

Unfortunately, the only response I got that even came close to putting a number on the value of EV certs demonstrated the one thing I already knew:

I didn't set out to write a long blog post, but here we are and I still feel like I've only just scratched the surface. For example, I haven't touched on how the certificate you see in the address bar tells you nothing about all the other sites you've just loaded active content like JavaScript off of - EV certs make no assurances of that. Nor did I delve into how the one technical upside EV proponents tend to clutch onto is about to become moot as Chrome begins requiring Certificate Transparency for all new certs in just a few months from now. I also didn't delve into how at present, 93% of the websites using HTTPS simply haven't seen the value in EV:

I want to finish by being crystal clear that I have no objection to the use of EV certs nor to the fact that they come at a financial cost and that commercial CAs profit by selling them. By all means, go and grab one if you think there's benefit because at the absolute worst, they're not going to do any harm and at best, some people may trust you more and that could translate into sales. I agree with many of the points made about visual indicator in browsers needing to adapt and consumer education being important and they're good goals, but they're not yet present day realities. The narrative around all of this has to be better and it has to be supported by those who don't have a vested interest in the commercial success of CAs, particularly in light of recent behaviour.

The bottom line is that as of today, the effectiveness of EV certs is entirely dependent on people recognising what they mean and actually adapting their behaviour accordingly. It's hard to argue with that.

Security SSL
Tweet Post Update Email RSS

Hi, I'm Troy Hunt, I write this blog, create courses for Pluralsight and am a Microsoft Regional Director and MVP who travels the world speaking at events and training technology professionals

Troy Hunt

Hi, I'm Troy Hunt, I write this blog, run "Have I Been Pwned" and am a Microsoft Regional Director and MVP who travels the world speaking at events and training technology professionals

Upcoming Events

I often run private workshops around these, here's upcoming events I'll be at:

Must Read

Don't have Pluralsight already? How about a 10 day free trial? That'll get you access to thousands of courses amongst which are dozens of my own including:

  1. OWASP Top 10 Web Application Security Risks for ASP.NET
  2. What Every Developer Must Know About HTTPS
  3. Hack Yourself First: How to go on the Cyber-Offense
  4. The Information Security Big Picture
  5. Ethical Hacking: Social Engineering
  6. Modernizing Your Websites with Azure Platform as a Service
  7. Introduction to Browser Security Headers
  8. Ethical Hacking: SQL Injection
  9. Web Security and the OWASP Top 10: The Big Picture
  10. Ethical Hacking: Hacking Web Applications