I had this idea out of nowhere the other day that I should have a visual display somewhere in my office showing how many active Have I Been Pwned (HIBP) subscribers I presently have. Why? I'm not sure exactly, it just seemed like a good idea at the time. Perhaps in this era of remoteness I just wanted something a little more... present. More tangible than occasionally running a SQL query. Or maybe I just wanted to geek out a little on some tech 😎 So I bought a LaMetric:
It's a little USB-powered display that connects via Wi-Fi and can pull data via a bunch of pre-configured apps (such as Twitter for your follower count) or you can write your own. For the HIBP subscriber count, all I needed was a number from the database but more importantly, I wanted to know as soon as possible each time it changed. There are only 2 events which can cause that number to fluctuate:
- A new subscriber verifies their email address
- An existing subscriber unsubscribes
Simple stuff, obviously, but what's the best approach to build this out? I often find the simplest solutions are the ones that can be built in the most possible different ways, so I thought I'd ask the masses:
Let's say you want to store a single integer in @Azure, update it based on certain events and read it from an API in @AzureFunctions. Where would you store that value? Is there a simple storage construct that'd work well?
— Troy Hunt (@troyhunt) January 22, 2021
The answer closest to what I was originally thinking was to use Azure's Durable Functions Entities. They'd be a fine choice, but there was a different approach suggested that really resonated with me:
It’s not staying purely inside Azure... but @Cloudflare workers with durable objects or just the KV store might be a very simple and performing way to wire it up.
— Sean W Richardson (@swr) January 22, 2021
It resonated because I realised I could build this out with almost no code at all in the origin services that run on Azure and instead delegate both the persistence of data and host the API itself all within Cloudflare. Here's how it works:
Firstly, the whole point of this exercise is to build a LaMetric indicator app (that link gives you a full walkthrough of the process). The key goal here as it relates to pulling data from a service is that there should be an API on the back end somewhere returning JSON like this:
{
"frames": [
{
"text": "0",
"icon": "i43168"
}
]
}The "text" is the information to be displayed on the device and the "icon" is what'll sit to the left of it like the Twitter one in the earlier photo. The text is defaulting to zero and the icon identifier was generated for me when I first went through the app setup process. So, all I need is an API to return data in the format above which brings me to Cloudflare Workers.
I've written about Workers before but as a super brief recap, they're code that runs "on the edge" or in other words, in each of Cloudflare's hundreds of reverse proxy nodes spread around the world:
That map is a little dated now as it's from the aforementioned blog post from a few years ago, but you get the idea. It's your code running on each of those purple dots, reading and manipulating incoming requests and outbound responses from your origin service. Per Sean's suggestion above, I decided to combine Workers with Cloudflare KV or "key-value":
Workers KV is a global, low-latency, key-value data store. It supports exceptionally high read volumes with low-latency, making it possible to build highly dynamic APIs and websites which respond as quickly as a cached static file would.
So far so good, any queries to it are going to be super-fast and no problems doing heaps of them, but here's the caveat:
KV achieves this performance by being eventually-consistent. Changes may take up to 60 seconds to propagate. Workers KV isn’t ideal for situations where you need support for atomic operations or where values must be read and written in a single transaction.
None of that worries me at all because it's a simple little informational display and the number is really of little consequence, so it doesn't matter if it's very slightly off for data accrued over the last minute. I decided to persist the subscriber count because by keeping it in KV, I could hit the service as frequently as I liked without those requests going back to Azure. And hey, so much of what I do on HIBP is completely transparent so why not let other people have access to this data too if they want? All I needed to do was to maintain that subscriber count in KV and have an API sit on top of it to return the value in the fashion the LaMetric app required.
Let's start with KV and you need a namespace to get up and running. I called mine "hibp":
Within that namespace I then created a key called "hibp-subscriber-count":
You're seeing the value as at the time of writing, I'll get into how I set that dynamically in a moment. For now, the important thing to understand is that I have a value I can persist and read from or write to from any Cloudflare edge node and it'll propagate across to the other edge nodes within about a minute ("eventual consistency").
But how am I going to write to the KV entry? Remember how earlier on I said the only time this value changes is when people either confirm their subscription or cancel it? I went into the HIBP code and added one simple line to the MVC controllers that return those responses:
HttpContext.Response.Headers.Add("hibp-subscriber-count", activeCount.ToString());The "activeCount" variable is returned by a separate method that simply counts all the active subscriptions in the DB. By putting it into a response header, it doesn't change the body content that's returned to the browser and it's easily accessible via a Cloudflare Worker like this:
addEventListener("fetch", event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
let response = await fetch(request)
let subscriberCount = response.headers.get("hibp-subscriber-count")
if(subscriberCount !== null){
await hibp.put("hibp-subscriber-count", subscriberCount)
console.log(subscriberCount)
}
return response
}If the "hibp-subscriber-count" response header exists, set that as the value of the "hibp-subscriber-count" KV entry and return the original response from the origin service. For this to work you do need to bind the KV namespace to the Worker via the Worker settings:
The only thing left to do now was to add routes for the 2 places the subscriber count changes. This ensures that when requests are made for those resources, the Worker is invoked:
The wildcard on the end of each URL represents the unique token which is passed to identify the subscriber. And that's it as far as updating the KV entry goes. There are occasions where one of these routes doesn't actually change the subscriber count (for example, if the subscription has already been verified or if the token is invalid), but in those cases there's no header in the response and the Worker code simply returns it without touching KV. Initially, I was using the Worker to strip out the header before it was returned to the client, but frankly, that's just unnecessary code. Who cares if the client has that header in it? It's public info by virtue of the API I was going to publish for LaMetric anyway. Speaking of which, here's what that looks like:
addEventListener("fetch", event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const value = await hibp.get("hibp-subscriber-count")
if (value === null) {
return new Response("No HIBP subscribers found", {status: 404})
}
const data = '{"frames":[{"text":"' + value + '","icon":"i43168"}]}'
return new Response(data, {
headers: {
"content-type": "application/json;charset=UTF-8"
}
})
}That's an entire API written as a Cloudflare Worker that only pulls data from KV. If it can't find the data then it returns 404 (which would break the LaMetric app), otherwise it's just some very basic JSON adhering to the spec from earlier and a content type header. I then added a route to make it accessible on the HIBP domain:
Because there's no auth involved, you can easily hit it yourself right here: https://haveibeenpwned.com/api/v3/subscribers/lametric
The last bit was to integrate that back into the LaMetric app which is all configured via a web UI (the "app" isn't like an app store app, it's more like an integration that surfaces itself via the inbuilt store within the LaMetric app):
If you have a LaMetric account, you can access the app in their portal (I've made the visibility public) and pull it down to your own LaMetric device. And if you do, it'll end up looking just like this:
I set a little audible notification sound on change just for fun (this is done within the LaMetric app on your iOS client), but with 1k to 2k new subscribers on an average day, the novelty is gonna wear off that real fast I think. This was a fun little project whipped up from start to finish in a couple of hours and whilst it doesn't really do anything of tangible value, it's kinda nice knowing that in this strange time where we all feel rather isolated, each time that number rolls over there's a real human somewhere in the world using a little piece of software I've written 🙂
Oh - one last thing - here's what it looks like via time lapse over most of today, pretty cool I reckon:
Here’s my @getlametric with custom @haveibeenpwned app tracking subscriber signups via @Cloudflare Workers using KV (time lapse over most of today with changing lights). Full blog post coming in a moment... 🙂 pic.twitter.com/CP8VxKCFic
— Troy Hunt (@troyhunt) January 25, 2021