You may not know this, but an HTTP 403 response when browsing to an empty directory is a serious security risk.
What the?! You mean if I go to my website which has a “scripts” folder where I put all my JavaScript and I have directory browsing disabled (as I rightly should) and the server returns a 403 “Forbidden” (which it rightly should), I’m putting my internet things at risks of being pwned?!
Yes, because it discloses the presence of a folder called “scripts” which is a common directory.
Well of course there’s a bloody folder called “scripts”, all my HTML source which you can see references it! I could call it “i-love-drunken-elephants” and you could still see it so what’s the point?!
But it would still return a 403 which would confirm the existence of the resource and pose a directory enumeration risk.
But you can discover the presence of the directories anyway! Ok, in today’s modern apps like ASP.NET MVC they might actually be routes that don’t translate through into physical paths but still, this is just being pedantic!
Your site can’t go live until you fix it.
Uh, let me just fix that for you…
Getting to grips with the underlying issue
This is one of those things that rightly or wrongly, I’ve seen popping up from various security teams and automated scanners in recent times. You can argue it all you want (and the severity of it is contentious), but the fact that it rears its’ head and causes debate is enough to just fix the damn thing and be done with it. Oh – and incidentally, I ran a Netsparker over Have I been pwned? (HIBP) recently and this was one of the findings so yeah, it affects me too (although I have the luxury of choosing to ignore it if I like!)
Let me show you why this happens: in the source of each page I have a <script> tag like this:
<script src="/scripts/pwned?v=2VUGdHR_7X4UImu2rpAgoquGkcvoIBUlzD35P5Y-dgo1"></script>
This is actually using ASP.NET bundling and minification to combine multiple scripts into one and then squish all the JavaScript, but what it means is that it’s implying there is a path which is simply “/scripts”. If we hit that path we’ll get the following:
Yes, I have custom errors configured for the app but they don’t catch the 403.14 returned when the user isn’t authorised to browse a directory with no default page present. Hang on – what’s the .14 bit? That’s the sub-status code that IIS returns for this particular flavour of a “forbidden” error. You don’t see the sub status code reflected externally in the response, but it exists within the process returning it.
You’ll find the same error being returned for a range of other paths that are easily discoverable including /content, /content/images and /fonts. Every time you have a directory, you run the risk of a 403.14 being returned and security people getting uppity.
One way you can address this is to create an incoming URL Rewrite rule such that every request for a known empty folder simply gets sent off to your default custom error or generic 404 page. That works, but it’s not scalable as you have to do it for every single folder including when you add them later on – which you’ll inevitably forget about.
No, something more sustainable is called for.
Catching the 403 response (and why you can’t)
My original thinking was this: I’ll create a URL Rewrite rule that catches the outbound 403 response and simply rewrites it to a 302 and appends a location header or in other words, tell the browser to redirect off to my usual 404 page. Despite how brilliant I thought this solution was, it just wouldn’t play ball. The response wasn’t being caught by the rule and the status code wasn’t being rewritten. In desperation, I turned to Stack Overflow and explained that I Can't change IIS response code with URL Rewrite outbound rule. I also emailed fellow MVP and URL Rewrite guru Scott Forsyth.
You can read Scott’s response on Stack Overflow but in short, that just wasn’t going to work. URL Rewrite can’t rewrite response codes so something else was called for. There are probably ways of tackling this with an HTTP module or somewhere within the lifecycle of the response, but that’s not a configuration-only solution and I really wanted to keep this constrained to something that can be done without recompiling, simply because that’s an important advantage for a lot of people, particularly when they’ve already got live running websites and they get a finding like this.
But the solution did start to unfold in Scott’s response and it all comes down to how errors are handled within system.webServer.
Defining custom errors in system.webServer (and how it’s only a partial solution)
Here’s the answer from Stack Overflow which on the surface of it, makes good sense:
<httpErrors errorMode="Custom"> <error statusCode="403" subStatusCode="14" path="/Error/PageNotFound" responseMode="ExecuteURL" /> </httpErrors>
Let’s now see how that looks on HIBP:
Ok, se we’re actually getting somewhere – kinda. We’re seeing “Page not found” and the request is returning an HTTP 200 which will satisfy the “We don’t want to see a 403” demands, but it’s not consistent with a normal 404 on the site. For example:
One could argue that whilst yes, there’s no longer a 403 and that particular checkbox can be ticked, the fact that the directory browsing error returns the “Page not found” page on the original URL with an HTTP 200 while the the genuine 404 redirects to /Error/PageNotFound effectively discloses the same thing as before – that there is a physical folder called “scripts”.
We can begin to fix this quite easily by changing the “responseMode” attribute to “Redirect” instead of “ExecuteUrl”. Here’s what happens now:
Ah, that’s better – kind of. See how we now have the same 302 response followed by a redirect to /Error/PageNotFound – that’s the good bit. The bad bit is that we still have a fundamental difference in that there’s no “aspxerrorpath” query string when the redirect happens system.webServer. You can’t add it either, not by configuration and nor can you remove it from the custom error which handles the genuine 404, at least not without same hackery.
Time to get creative.
Removing the aspxerrorpath query string
There’s actually a very simple solution to removing the query string from your custom errors configuration and it’s this:
<customErrors mode="On" defaultRedirect="~/Error"> <error statusCode="404" redirect="/Error/PageNotFound?foo=bar" /> </customErrors>
Which then means the /foo path from earlier gives you this:
That might look a bit screwy (and it is), but you can now go and apply the same query string to the redirect in system.webServer and wether it’s a genuine 404 or it’s a actually a 403.14, you’ll get the same screwy query string – job done! It doesn’t matter what the query string is called (“foo” is merely coincidental) and indeed you don’t even need a name value pair, you can just append the question mark and that alone is sufficient.
The problem, however, is that I don’t like screwy query strings appearing in the response for no (apparent) good reason so we need a bit more hacking away yet…
Rewriting the redirect location
The problem now is that I want to take /Error/PageNotFound?foo=bar and get rid of the query string. In case the semantics of redirects are not entirely familiar, when a web server responds with a 302 like in the screen grab above, you also get a “location” header which looks like this:
Location: /Error/PageNotFound?foo=bar
This tells the browser to issue a subsequent request to this path which is why you then see the second request above. This is what we need to change and fortunately it’s an easy fix with URL Rewrite. It looks just like this:
<outboundRules> <rule name="Change location header" patternSyntax="ExactMatch"> <match serverVariable="RESPONSE_location" pattern="/Error/PageNotFound?foo=bar" /> <action type="Rewrite" value="/Error/PageNotFound" /> </rule> </outboundRules>
Pretty simple stuff – take the “location” response header when it’s trying to redirect the browser to our screwy query string path and then just strip it off. Let’s try it now:
Hey, this looks alright, now it’s exactly the same as the behaviour when you load the scripts path! All it took was a custom error in system.webServer then a screwy query string in the system.web custom error for the 404 response and a URL Rewrite rule to change the location. How simple is that…?!
Ok, I’m being slightly facetious because it should be easier than that, but it’s not. On the plus side though, here’s what we now have:
- There is no longer a 403 returned when browsing a directory with no default page when directory browsing is disabled.
- The response from a genuine 404 on a non-existent path is identical to browsing a physical path with no default doc.
- It’s all done by configuration only, no code.
So it’s perfect then? Well kinda, all except for one little thing…
Circumventing courtesy redirects
If you’re eagle-eyed enough, you may have noticed a very slight difference in the way I requested foo and scripts earlier on with the former being referred to as “/foo” and the latter “/scripts/”. That trailing slash makes a big difference because here’s what happens when it doesn’t exist:
What madness is this?! A 301 “permanent” redirect followed by a 302 “temporary” redirect and finally the “PageNotFound” page – what gives? It’s a courtesy redirect, that is IIS is actually looking for a file called “scripts” and when it can’t find it, realises that’s there’s a directory with the name and spits back a 301 telling the browser very explicitly that this is indeed a folder by virtue of the newly appended trailing slash. Of course it also tells Mr Hacker the same thing so whilst the 403 is gone and the paths and the query strings are all good, that extra redirect gives the game away.
To be fair, as of about right now you’ll no doubt get a tick from security folks as there is no longer a 403. If that’s your objective then you can stop here, in fact you could have stopped right after the system.webServer custom error entry but IMHO, it kind of feels half-arsed. If I’m going to do this, I’m going to do it properly dammit!
I went backwards and forwards a bit with Scott on this until we came to an implementation which looks like this:
<system.webServer> <defaultDocument enabled="false" /> </system.webServer>
This is pretty self-explanatory – disable the default document capability. What this then means is that the DefaultDocumentModule (that’s right, no spaces, it’s a thing) no longer causes the 301 to the path with the trailing slash in order to imply the request is to a folder which should then serve the default document. It means the request now looks like this:
It also means each of the following scenarios responds identically:
GET https://haveibeenpwned.com/PathDoesntExist HTTP 302 -> /Error/PageNotFoundGET https://haveibeenpwned.com/PathDoesntExistWithTrailingSlash/
HTTP 302 -> /Error/PageNotFoundGET https://haveibeenpwned.com/scripts
HTTP 302 -> /Error/PageNotFoundGET https://haveibeenpwned.com/scripts/
HTTP 302 -> /Error/PageNotFound
Hallelujah – it works!
Except for the times it doesn’t. Actually it always works on HIBP, but that’s because it’s an MVC app that implements routing to map a URL such as https://haveibeenpwned.com/About to the “About” controller. If it was a Web Forms app and it depended on a default.aspx file in the “About” directory then I’d have a whole new problem.
However, there are multiple avenues to address this. One is to just enable default documents for a whitelist of paths. Ok, not very scalable but certainly feasible.
Another is to use URL Rewrite to redirect queries to specific paths that rely on a default document to the URL that explicitly contains the file name in it, for example “/foo/default.aspx”. Again, not real scalable and also not real pretty.
A better option would be to use URL rewriting or in other words, set the system.webServer response mode to “ExecuteURL” and the custom errors redirect mode to “ResponseRewrite”. This will then show the error page on the requested URL without any redirecting whatsoever. If you’re worried about the SEO implications of an HTTP 200 showing an error page, you can always change the response code in the error page itself.
Again, it’s more hackery, but that last option is probably your best bet if you’ve got a dependency on the presence of default documents in folders.
Summary
If I’m honest, all of this is a lot of mucking around for very little end benefit. Less charitable people than me would call it “security theatre” and in the spectrum of potentially exploitable risks, this is way, way down the bottom.
But none of that changes the fact that security tools and teams view this as a risk and it raises a flag and you need to fix it. That ultimately it can be fixed very easily (but only because you now have the path laid out for you!) and via configuration only is a good enough reason just to do it and move on.
Oh, and in case you have a more efficient means of doing this either by configuration or code, do leave a comment with some tips. This is the sort of thing people inevitably Google their way into and the easier we can make life on those who follow, the better.
Just to make it real easy, here’s all the config in one go:
<system.web> <customErrors> <error statusCode="404" redirect="/Error/PageNotFound?foo=bar" /> </customErrors> </system.web> <system.webServer> <rewrite> <outboundRules> <rule name="Change location header" patternSyntax="ExactMatch"> <match serverVariable="RESPONSE_location" pattern="/Error/PageNotFound?foo=bar" /> <action type="Rewrite" value="/Error/PageNotFound" /> </rule> </outboundRules> </rewrite> <httpErrors errorMode="Custom"> <error statusCode="403" subStatusCode="14" path="/Error/PageNotFound" responseMode="Redirect" /> </httpErrors> <defaultDocument enabled="false" /> </system.webServer>
There, don’t you feel more secure now? :)