Mastodon

IoT Unravelled Part 1: It's a Mess... But Then There's Home Assistant

23 November 2020

With the benefit of hindsight, this was a naïve question:

In my mind, the answer would be simple: "Just buy X, plug it in and you're good to go". Instead, I found myself heading down the rabbit hole into a world of soldering, custom firmware and community-driven home automation kits. Finally, a full 123 days later, I managed to open my garage door with an app:

What follows is my journey down that very deep, very dark rabbit hole from which I thought I'd never emerge. Every time I thought I had an answer, it raised 2 more questions. I had to buy new gear, learn new acronyms and work with things I'd never heard of before. I waded through endless online discussions with strong opinions expressed in frequently conflicting directions about things I thought would be simple. Never before have I headed down a technology path that, frankly, is such a fragmented mess.

I added to this blog post as I progressed with a view to ultimately having a "happy path" for others to follow in the future. Instead of a single post, I ended up with a 5-part epic which I'll post piece by piece over the course of this week with the hope that others can follow along and eventually, with enough time and patience, be able to open their garage doors too.

Everybody Wants to be "The IoT Solution"

This is where the whole mess starts: what is the hub of your IoT world? In my original tweet above, I thought it would be Apple's HomeKit which seemed like a reasonable assumption given my family's dependencies on iPhones, iPads, Apple Watches and with an Apple TV in the house. Those in an Android world might reasonably assume that Google Home would be their hub. Others might assume it'd be something Alexa related. But no, every one of those answers is wrong because every single one is a proprietary ecosystem with fragmented support by different devices and a kludge of vendor lock-in. Take this as an example:

Now firstly, the Nerf gun wall is freakin' epic! But secondly, that Arlec LED strip Ari is attaching to the back of the wall runs on the Grid Connect ecosystem which includes support for:

Google and Amazon. That is all. No HomeKit. In fact, HomeKit was often the odd one out when I looked at IoT products and I can imagine someone explaining it away as Apple demanding things be done the Apple way or the highway hence the lack of compatible devices. And this wasn't a typo on the Grid Connect website:

It's not just the big players either; you'll find all sorts of lesser-known brands wanting to be the hub of your IoT world. During this whole exercise, I decided I needed to replace the receiver in my home entertainment setup as it wasn't powerful enough to drive the speakers I have. So, I'm in at the home theatre shop and I see this beautifully made universal remote control:

This lovely, brushed aluminium unit is made by Control4 and guess what? They can be the hub of your smart home! Oh - but it's not self-configurable and you need a licensed installer to set it all up for you 🤦‍♂️

Which brings us to Home Assistant or for the sake of brevity, HA.

Home Assistant

This feels like a fair place to start:

But it's also slightly disingenuous because whilst on the surface it may look like yet another solution to the same problem, it's philosophically different in several key ways:

Open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.

The Apple / Google / Amazon solutions are all proprietary and tied very closely to the respective tech behemoths' commercial offerings. They want people on their platforms, using their clouds and buying their products. HA, on the other hand, doesn't care who made what or which devices you have or whose clouds you're on, it just wants to tie it all together in a meaningful way. It's an extremely active open source project typically (but not exclusively) run on a Raspberry Pi with a goal of integrating all of your things and keeping them self-contained within your own home rather than being dependent on public cloud services. It has amazing community support and a very devoted fan base which has helped catapult it into one of the top open source projects on GitHub. However...

The learning curve is steep, it has a bunch of rough edges (it's still not reached a v1 as of the time of writing), you end up living in YAML and by any reasonable measure, it's only usable by geeks who are happy living in a world unfamiliar to most mere mortals. Fortunately, that includes me and despite the "maker" nature of the whole thing, I'm massively impressed with HA and nothing else I've seen along my IoT journey comes even close to comparing. If you're going to do IoT in any meaningful way, you start with HA.

The thing is though, you need to be ready to get your hands really dirty:

I captured this tweet and dropped it into the draft blog post as I was lamenting just how damn hard it was to make simple things work the way I wanted them to. But I don't want to get anywhere near that level of detail in this blog series as it'll just scare people off, let me instead focus on the basics and provide enough background to get people heading in the right direction, starting with the fundamental principles of what makes HA great.

Integrations, Devices, Entities, Automations and Scenes

When I'm explaining this to people, I put it like this:

HA integrates with other devices and services which it can both use as triggers and perform actions on.

For example, just to jump straight to the conclusion for a moment, I now have a bunch of little motion sensors around the house that can turn lights on:

This is achieved firstly, by using an integration that can communicate with the sensors and in this case, it's deCONZ which is a product under the Phoscon brand made by Dresden Elektronik (hence the "de" in deCONZ):

I've left other integrations in the screen cap above to give you a sense of some of the other things HA can communicate with: Alexa, WeMo switches, Elgato key lights and many other connected devices in my house. At the time of writing, there are 1,713 different integrations including... a Have I Been Pwned Integration!

The deCONZ integration enables communication with Zigbee devices (more on that in part 2) and per the screen cap above, I presently have 35 of those in my house. One of them is the first motion sensor in the earlier tweet which, in HA, looks like this:

The motion sensor is a device I placed at the bottom of the stairs in my house which spans 3 floors. I have another motion sensor halfway along near the kids' room and another again at the top near our master bedroom. These motion sensor devices each have 3 different entities:

  1. The amount of light they can currently see
  2. A battery level
  3. Whether or not there's currently motion detected

This nomenclature threw me a bit at first but it makes sense: one physical device you can hold in your hand may measure many different things and each one of those things is considered to be an entity (entities aren't just about measurement but let's use that as a simplistic example for the moment). I have little temperature sensors in each room and each one of those devices can measure humidity, pressure, temperature and has a battery state:

Yet another device I now have all over the house is an IoT relay called a Shelly, two of which you can see in the tweet below (they're the little blue units amongst all the wires):

Yes, this is pretty normal Aussie wall socket wiring. No, we don't use gang boxes. Yes, I'm conscious you do things differently in other parts of the world. Moving on, each Shelly has a single entity which is simply a power switch:

So, now we have all the mechanics required to tie together automations and as you can see in the screen cap above (and in the earlier one that shows the stairs motion sensor), I have 2 automations using these devices. I'm not going to go anywhere near the YAML involved in this blog series, let's instead focus on the logic:

  1. If motion is detected at the bottom, middle or top of the stairs and the light level down the bottom is beneath 200 lumens, turn the Shelly on the light switch on
  2. If all 3 motion sensors haven't detected any motion for the last 5 minutes, turn the Shelly on the light switch off

And that's it 🙂

But that automation just turns on a single light, what if I wanted to turn on more lights? Or play music? Or set other device statuses, all at the same time? This is where we get to "scenes" which allow you to define multiple pre-set states that can all be switched on in one go. For example, Ari wanted a fast way to get all his lights back to the same state he referred to as "cool", so we made a scene that turned on a bunch of lights and set the nerf gun wall to red:

That scene can be trigged by various events, including automations. We'll come back and look at that again shortly but for now, let's talk about how all those lights work in the first place.

Interfacing with Devices: Different Brands, Same Stuff

Ever get an odd sense of déjà vu when you see a product under a certain brand and you could swear you'd seen it somewhere before, but just different? I had that recently when I drove a friend's Tesla Model X and the indicator stalk felt just like the one in my Mercedes. Turns out that because it is just like the one in my Mercedes with the US brand using a bunch of parts from the Germans.

And so it is with IoT bits too:

Earlier on I mentioned I'd bought the Grid Connect lights (which are manufactured by Arlec) to put behind Ari's Nerf Gun wall. They were cheap (at least compared to something like Philips Hue), RGB (not just different shades of white) and could talk over Wi-Fi (not just controllable via a remote). The Grid Connect app worked fine but as I also mentioned earlier, there's no way it'd work with Apple's HomeKit and based on feedback to the tweet above, nor were there any integrations with HA. But there was another option:

Ok, that's actually several different options but let's just focus on Tuya first because that's the easiest path. Tuya is "the World's Leading IoT platform" (yay, another platform 🤦‍♂️) and per the above tweet, they ship products that run a ESP8266 chip which is a pretty common piece of kit. Consequently, I could pair that Grid Connect light strip with the Tuya app which... then says Arlec!

Later on, I bought a heap of RGB downlights from Oz Smart Things:

Turns out these are also Tuya compatible so now, without directly buying a single Tuya product, I have a lot of products running in the Tuya app:

What that means is that it's dead easy to control things such as the colour and the brightness:

Now, let's bring it back to HA for a moment and the value proposition here is that per Chris' earlier tweet, there's an integration that can bring these devices right into the same environment all my other IoT things are now in:

Ok, so far so good, now let's get to the twists in all this starting with how the same device looks in HA:

That looks fine now, but when I first added the downlights, I had no colour control. Power and brightness, yes, but no colour. It took some researching to realise that Tuya had killed colour control for a whole bunch of other people too and that the answer was to edit the customize.yaml file and for each Tuya entity, change the supported features to indicate colour control. I'm not even going to get into the mechanics of that here because that's not really the point of this series, rather I want to highlight how I kept running into "compatible but not completely compatible" scenarios like this.

So, now should I change the colour via the Tuya app or the HA website or app? In one way, it doesn't matter because the state is reflected the same in both (i.e. make it red in HA and the Tuya app shows it as red). In another way, I'm now in this vicious cycle of having more and more apps for specific IoT devices that whilst integrated with HA, still require the OEM app to do a heap of stuff. For example, the Nanoleaf app has a very specific set of controls:

These allow predefined scenes to be played on the light panels (something that can also be triggered by HA), but also very specific configuration such as what colour to make individual panels. AFAIK, that capability isn't built into the HA integration and even if it was, it's unlikely the experience in setting it up would rival that of the dedicated app.

The bottom line is that you inevitably end up with multiple different interfaces into the same device be they native interfaces provided by the device manufacturer or those exposed via the HA integration. But it doesn't end there either, because you still need to get those devices back into the other IoT hubs. Here's why:

Integration with Other IoT Hubs

Earlier on, I lamented that everyone wants to be the hub of your IoT world and that fortunately, HA can play that role in the place of one of the large incumbent tech companies. Mostly. The gap comes when you start using products from one of those companies that expect to communicate with IoT devices via their own proprietary product. For example, both my kids have an Amazon Echo Dot in their room. I put these in there in part because they're fun learning devices they can easily ask questions of (they can also ask "Alexa, who's Troy Hunt" and get an answer or, as I learned last night with my daughter, they can ask "Alexa, is Troy Hunt handsome" and get a resounding "He is handsome" 🤣), and in part to control their IoT-enabled devices. The latter can be done without even needing HA as Tuya natively integrates with Alexa and exposes all the devices:

But there are other things in HA which we might still want to access via Alexa, for example the "cool" scene I set up for Ari earlier on. To do that we need to surface data from HA into Alexa which can be done with the Amazon Alexa integration. Once configured, the scene appears and he can talk to the Echo Dot and ask Alexa to "turn on cool in Ari's room":

Just as there are reasons to surface HA into Alexa, there are also reasons to surface HA into Apple's HomeKit, a feat that can be achieved with the HomeKit Bridge integration. (Note: there is also Homebridge which is a different beast altogether.) We can now surface all the same devices into Apple's ecosystem:

Remember, that Nerf gun wall is running the Arlec LED strips that use Grid Connect surfaced through the Tuya app which all has no support for HomeKit yet here we are with control in HomeKit 😎

That extends all the way to Apple Watches too:

One of the primary reasons for doing this is that once you're in the Apple ecosystem you have Siri so now I can just raise my wrist and say "Hey Siri, open the garage door". Or if I'm in my car with Apple CarPlay I can issue the same command without even taking my hands off the wheel.

This does, however, create other problems especially when it comes to troubleshooting. Take a look at the hero image at the top of this blog post. Nice Nerf gun wall representing Ari's room as HomeKit sees it on my iPad, except... his desk power is showing "No Response". Huh, wonder if he accidentally unplugged it this morning? So I go into his room and no, he hasn't unplugged it because his neon-backlit keyboard is glowing and it's plugged into the USB hub connected to the TP-Link HS100 smart plug that's presently unresponsive. It's surfaced in HomeKit via the HomeKit Bridge integration in HA so I go into that device and the entity is disabled. If HA can't see it then HomeKit won't be able to see it. Ok, let's go further up the stack and now I'm in the native iOS Kasa app where the plug is both present and responsive to power on / power off commands which means it's on the network just fine. I stopped there because frankly, I got a bit pissed with the whole thing and just want to finish this blog post right now, but I've included this here to demonstrate just how many moving parts are required to make all this work. Just one of those moving parts stops and not only does it kill a part of your internet of things, but there's a good chance you'll be in for a lengthy troubleshooting session.

This all took me a while to wrap my head around, namely the fact that you can't escape the necessity to have multiple "hubs of all your things" and that they can all work together harmoniously... most of the time. It's not difficult, but it is fiddly especially when you end up with the same devices in HA as Alexa can natively talk to then you see dupes or you get to HomeKit and there's a zillion devices in there when you really only want a couple of lights. There are solutions to these problems, however, it just requires a little patience and a lot of tweaking.

Summary

I want to wrap up part 1 here because it's a nice  finish before peeling back more layers of complexity. Get yourself a Raspberry Pi, install HA and add integrations for your existing devices, surface through your platforms of choice and you're off and running. But that's only the first step, in part 2 I want to start delving into the implications of all those IP addresses, the role of Zigbee, custom firmware and soldering and ultimately, finding my own happy path to how complex I wanted to make the whole thing.

Lastly, I'm going to devote my regular weekly update live stream to this blog series and do it in conjunction with my good mate Scott Helme whose been going through his own IoT journey. It'll be broadcast at 17:00 Gold Coast time for me this Friday which is 08:00 the same day in London and 23:00 Thursday evening on the US west coast. If you can't make the live stream, it'll still be available here afterwards:

  1. Part 1: It's a Mess... But Then There's Home Assistant
  2. Part 2: IP Addresses, Network, Zigbee, Custom Firmware and Soldering
  3. Part 3: Security
  4. Part 4: Making it All Work for Humans
  5. Part 5: Practical Use Case Videos
IoT
Tweet Post Update Email RSS

Hi, I'm Troy Hunt, I write this blog, create courses for Pluralsight and am a Microsoft Regional Director and MVP who travels the world speaking at events and training technology professionals

Troy Hunt

Hi, I'm Troy Hunt, I write this blog, run "Have I Been Pwned" and am a Microsoft Regional Director and MVP who travels the world speaking at events and training technology professionals

Upcoming Events

I often run private workshops around these, here's upcoming events I'll be at:

Must Read

Don't have Pluralsight already? How about a 10 day free trial? That'll get you access to thousands of courses amongst which are dozens of my own including:

  1. OWASP Top 10 Web Application Security Risks for ASP.NET
  2. What Every Developer Must Know About HTTPS
  3. Hack Yourself First: How to go on the Cyber-Offense
  4. The Information Security Big Picture
  5. Ethical Hacking: Social Engineering
  6. Modernizing Your Websites with Azure Platform as a Service
  7. Introduction to Browser Security Headers
  8. Ethical Hacking: SQL Injection
  9. Web Security and the OWASP Top 10: The Big Picture
  10. Ethical Hacking: Hacking Web Applications