HA01 The Nerd's Ultimate Guide to Reasonable Home Automation (WIP -- early draft -- thoughts welcome!)

*Note the following writeup is at best a draft status. I would love to hear from those in the community that have done, or are doing, this. At some future date I want to turn this into a video, but I am using this writeup to organize my thoughts. Much to my surprise, there is a lot of good documentation out there but not a lot of the dots are connected for new-comers. Worse yet I see a lot of 'not invented here' that makes accomplishing something reasonable unnecessarily difficult because other guide authors shy away from ready-made commercial solutions e.g. for sensors or outlet control.

HA02: Ha02 - Getting started with a DSC Home Alarm
Better IoT: DSC Alarms w/Home Assistant Home Automation - YouTube

HA03: Coming soon

HA04: HA04 - How-To setup Home Automation Smart Plugs -- Flashing the S31 on Linux from Sonoff

HA05: HA05 - Adding Z-Wave to Home Assistant

Assume I know nothing. Educate me. Let me know what you have first hand experience with, and what works for you. *

Getting Started – Goals

I am aware of commercial systems and I have direct (computer janitor level) experience with many of them including OnQ, Vivint, Wink and several others. In my view these systems are much more of a grift than something actually usable.

A commercial system that does what we want does not exist. An open source single-vendor system does not exist for home automation.

It is easy to get caught up in the novelty “gee-whiz” aspect of that. If you know me through my other work, you might be thinking “oh no” but in reality I’m pretty practical. The philosophy we’re after here is the best technology gets out of the way.

The technology on star trek: tng wasn’t cool because it was the over the top and in your face (looking at you, st: discovery season 3) it was cool because it was really good at reading your intent and getting out of the way.

Do you really need a tiny LCD screen everywhere you currently have a light switch? For me, if you want to replace all my light switches, you’re going to need to meet my expectations of something just as functional and reliable as a spring and some bits of metal while also offering me a new compelling feature that I actually want.

If you want the gimmicky stuff, this is not the guide for you.

What do I actually want? And what I intend to share in this guide…

Here are some features I’ve already built into my smart home:

  • A kickass burglar alarm with lots of motion sensors

  • The low-voltage under-cabinet LED lighting in the kitchen comes on in the kitchen only if it is dark enough outside and there has been motion in the kitchen.

  • Rooms that have motion sensors will auto shut-off the lights if no motion is detected

  • If someone turns the light back on within 5 minutes of the auto-shutoff, the no-motion timeout is automatically doubled.

  • Outdoor lighting activates automatically when the phone comes in range of the wifi after not having been in range, assuming it is between dusk and dawn.

  • The basement sump pump is monitored. A second warnign sensor that the sump pump is not keeping up is in place.

  • The boiler uses hard-wired honeywell sensors for temperature range, pressure range, water level (redundant) for safe operation. It has been certified for safety. In addition if any z-wave or hard wired smoke detector goes off, it kills the natural gas outside at the meter unless a programmable output is activated at a nearby alarm panel within 1 minute.

  • any PIR motion detected on the front porch is logged such that it’s easy to pull up the video footage later.

  • same with doorbell

  • when the mailbox is opened at the end of the drive, it wirelessly reports the event.

  • o2 levels are monitored and controlled via raspberry pi wired up to a fresh air exchanger in the basement (simple relay control, industry standard). O2 levels are logged via serial port. Can function without Pi but can be overridden by Pi gpio output.

  • power draw and duty cycle of two heat pumps is monitored via the breaker panel. This already detected a short-cycling fault condition in one heatpump – the problem was a faulty pressure control valve that only fires in winter, not summer, on this particular model. The heatpump was “working” but the short cycling is hard on the compressor and likely would have resulted in a substantial repair bill later. The faulty valve itself was only a $45 part. Big win.

  • no motion in the bathroom, but the pex manifold serving the bathroom has running water? Shuts the water main off and notifies me.

  • dishwasher is using water outside its usual operating parameters? water main shut off, notice sent.

  • same for pex leg serving the toilets

  • other misc burglar alarm integrations

  • guest wifi sends the guest to a web page with limited access to surveillance cameras, plex server link, public nas directory. Working on some type of automatic access to other stuff like how to get at the chromecast where they are, and audio? I only have the one guest bedroom but maybe also this for the living room. Still working on this.

What Still Needs Improvement

would love ideas/feedback for Actually Useful Stuff

I want an easier way to add stuff to the shopping list from the kitchen. That doesn’t necessarily involve taking my phone out, and Alexa is too handsy and also cloud-dependent. Often I will look at the ingredients I have leftover from something and think “if I had ____ I could use the rest of this for satisfactory meal _______” I like the idea of, and have tried, services like blue apron where you just pick meals and they do it for you? But in practice it costs more than it needs to and… can’t I do this with an API cheaper? I mean I already do exactly this via API for ebay, newegg, and amazon purchases… food prep is surely not a lot different?

I want to automate the micro of food mangement prep, keep track of expiration dates, and not have it be as it is now where I fall into fixing myself the same 12 meals over and over again. Variety is the spice of life!

I am going to put some soft LED lights in the main bathroom and maybe under the bed. If motion is detected, and it’s late ate night, these can come on very gently so I can see to get to the bathroom, see in the bathroom, and see on the way back. But with very very low light so my eyes don’t adjust.

I am planning a modified float switch in the main waste stack, out to the sewer. I’d like an early warning sign of some type if the main drain is struggling or moving slowly. We did a news story on a japanese bidet that could monitor your health via poop. This would be the perfect spot for that as well! Assuming I could work that out with the company. LOL

As I am using HomeAssistant for some of this, I think it would be useful to have “something” in home assistant for maintenancy tasks.

  • Did you change the air filters due on ___?
  • Did you flush clean water into the boiler due on ____?
  • Did you complete a walk test for the alarm yearly due on ____?
  • Did you clean out the gutters this year?
  • Did you put fresh bleach tabs in all the AC drains so we don’t get any backed up condensate lines?
  • Other Stuff I forgot?

I don’t want to be bothered to think about that and trust in that the machine is keeping up with it. One year I put bleach tabs in 3 times because I kept forgetting I’d already done it.

The HomeAssistant, Poe, can handle keeping up with all that stuff and I can check it off when it’s done if I do it.

What do I need to know to do this?

First, you need a home alarm. You might not want a home alarm, but many home alarm systems are the best option when it comes to monitoring, control, reliability and sensors. Motion sensors, glass-break sensors, smoke detectors, water-level detectors – DSC and Honeywell are the top two names that come to mind.

I can do a video showing off both a honeywell and DSC system. These are the bedrock foundation of any automation you might do. They support robust wired and wireless sensors (prefer wired sensors any time you can!), have robust panel software, operate without the cloud and are easy to interface with other things. HomeAssistant supports both alarm panels.

ADT, and to a lesser extent Vivint, have been DSC resellers since the dawn of time. Only in the last 5-10 years have these companies upped their grift game to try to lock you into their system or panels, or keep you locked out of reprogramming your own system.

DSC and Honeywell both, on the other hand, have built a robust and reliable system you can count on (And extend!).

Let’s take a look at an olderish DSC 832 panel.

TODO

The best option for DSC panel integration is the Envisalink V4, which will give you an ethernet interface. I strongly suggest you put this on your IoT subnet as this ethernet interface has tissuepaper-thin security. This interface can be used for cost-effective ($10/mo or less) alarm monitoring services via the internet.

Our panel here is configured with a cellular interface module (sim card paid for in monthly monitoring subscription) but can use the Enviaslink v4 as a backup option should the cellular be out for some reason.

Of course all of these panels also support a hard-wired analog telephone as well.

Here’s an example setup with the DSC panel:

TODO

It has 8 hard-wired zone inputs. That is, you can use one (or more) sensors on each of these inputs and the sensors will notify the panel of sensor events. This is a little more sophisticated than Normally Open or Normally Closed contacts – the alarm panel checks for the presence of an end-of-line resistor so if there is a dead short or a cut wire, the alarm panel will know. This will set off the alarm. Even tapping into the live sensor wire to try to fool the alarm panel is not likely to be successful.

You can add additional “zone expanders” for more inputs as well. If you have a wireless receiver, you can enroll that module in the panel and add wireless zones up to the zone limit of the alarm panel – typically up to 32 or 64 total inputs.

Good candidates for wireless zones are window sensors, door sensors and motion sensors. Wireless signals can be jammed, so use wired sensors everywhere you can.

I have also added two smoke detectors here, hard wired into the alarm panel.

The panel has a lot of configuration options. You should read up on all the programming options for DSC panels. It isn’t too hard once you get the hang of it.

The two smoke detectors are configured for 24/7 monitoring. If they trip, it will notify the alarm monitoring company, me and (optionally) set some other systems in motion that will cut off the natural gas to the house. Strictly speaking the natural gas cutoff is NOT hard wired, but it could be, into the panel.

This particular panel supports 4 programmable digital outputs, one of which is rated to drive a relay. In the event of smoke detector events, you go from 0 to 1 on programmable output 1. That’s enough to trigger the natural gas safety shutoff.

In my case, I’m not that worried about it, so I have a simple soft relay the natural gas company approved on the meter that’s more like ‘emergency stop’ aux input. I just close a wire and the meter itself turns off the gas. The gas company can even do it remotely! Hope no hacker figures that out :-\

Do all zones have to trigger an alarm?

No. This is the benefit of something with decades of battle-hardening and real-world use cases. There are “recipes” for these panels for monitoring water in a basement, sump pump operation, mixed zones, one panel for multiple businesses with a shared entry way, you name it.

So in the case of “motion sensor in the mailbox” I mentioned earlier – one setup possibility is this: Set the sensor in partition 1 and partition 2. The sensor will only trigger an alarm if both partition 1 in the system and partition 2 are armed. With the hypothetical setup here, only partition one is ever armed (or disarmed). You can go further and enable always-on “Door Chime” for that wireless zone. And most keypad will let you pick from a few preset chimes. So you could get a little “chirp” from the keypad when the mailman opens your mailbox.

This little feature was “free,” basically, just by having a completely industry-standard alarm AND also understanding exactly how it works and what features it has. Time is valuable! no need to DIY it with a raspberry pi stuck to your mailbox with bubble gum.

Some clever person literally designed this to work for this use-case somewhere and this is exactly the kind of thing some engineer inside Google completely forgot when designing his Nest Doorbell Ring Camera Mailbox iMail gateway thingamajig.

In this setup, the mailman can open your mailbox when your alarm is armed, and the alarm does not trigger.

Further automation is possible via something like HomeAssistant because HA integrates with both the DSC and Honeywell panels that I’m familiar with.

(Note DSC does make one panel that is The Devil. Avoid that one. It’s the Neo. Neo as in new, as in, oh man we didn’t realize we were supposed to grift our customers like all these other companies are doing. Our bad. Let’s try that again! Tada, here’s The Neo. Trust me – just take the blue pill on that one. Yeesh.)

The Zones can be re-labeled in your alarm keypad to be descriptive of what they are.

Wiring the panels is easy – uses a cheap small and inconspicuous 4-conductor somewhat-twisted pair wire.

How’d you achieve automatic under-cabinet lighting control?

Okay, so the alarm panel doesn’t do that. Alarm panel companies tried to get into the smart home business, but their solutions are dumb and don’t have the Thousand of Hours of Thinking that all the other things do, like motion sensors. (Seriously, there are a few different types of motion sensors and the DSC sensors let you configure how big your pet is, how big your room is, etc via jumper. Jumper! Don’t reinvent the wheel. Trust me. )

Ryan did this cool video TODO Link Here on Software Defined Radio and re-using old alarm parts. That’s part of the magic here. See we can use the alarm panel itself, and it’s sensors, as part of our automation.

Kitchen Alarm Sensor sees motion, panel triggers, Home Assistant knows (within 1 second!). Then HA can trigger a smart plug that turns on the under-cabinet lighting controller. That’s it.

It this system ever melts I can just plug it directly in, or control it another way, but it’s literally just a smart switch.

What smart switch?

I’ve tried several

TODO

This is the best one – inside it has an ESP3266. This is not great because this micro has had some security issues in the past. It is programmable, though, via FTDI. Check out the video on that.

I’d prefer a hard-wired version of this switch; I will probably do a video at some point on stripping out the ESP3266 entirely. I should do a two-wire bus module that is pin compatible with the ESP3266 so we can have some sort of reasonable 10-year security solution.

Home Assistant

The bedrock layer of the home automation, imho, is the alarm panel. But, by itself, it’s not all that magical.

Enter Home Assistant. It can be magical, but it’s a 90% solution. It’s like showing up to Lowe’s and asking “Where Is The House You Can Buy?” Well you can probably build a house from what Lowe’s sells, but you don’t buy ready-made homes from lowes.

That’s the key takeaway from this guide – you are learning enough craftsmanship here to build what you need – to build what makes sense. As a buyer of this technology, anyone that is selling you anything else, to some degree, is overselling you. They’re grifting you at worst. Even the best solutions only have 5-ish year useful lifetimes.

These alarm panels are designed for 20+ year lifetimes with just minimal maintenance (swap the industry-standard backup battery after 5-10 years), as are most of the rest of the other systems in your home. The electrical system is designed for 100+ years.

Some of the rest of the magic in here are so-called “Smart” breakers in your breaker panel. this will control all of the outlets on a given breaker, so that might be good or bad depending on what you want to do. I’ve found this useful for HVAC monitoring and outside light control. Often these breakers will still send a small voltage when they are “off” so that if you toggle a light switch in the chain, it will restore power to the circuit. Neat!

TODO show that

TODO show more about home assistant, add links

The solution for the cabinet lighting actually ends up being very simple. Once you have the plug (which is already supported by HomeAssistant) enrolled, and talking to your alarm panel (also supported by home assistant!) then it’s basically “when motion, turn light on” and then you can expand it from there. when motion, and it’s near dusk or not after dawn" and other rules that HomeAssistant already has built-in.

Once these rules are set, they will operate to your liking for many years.

One thing I like about HA is that they make backing up and restoring your configuration fast and easy.

It’s possible to net-boot a newer RaspberryPI and run HA on that. That’s potentially a HighlyAvailable Home Assitant. HAHA. Get it. HAHA? Haha.

I kind of want to demo that for the video. RPi primary went down. Kill PoE to that port. Activate secondary PoE port with backup Pi. HA restored to normal operation (Via netboot image). Wonder if I could talk to the HA people about an active-active cluster of RPi LOL. I digress…

TODO Show scripting this from Linux CLI (really easy to do with engenius’ switch api)

TODO More adventures with Home Assitant

TODO show enabling ssh that was a pita on rpi HA

TODO show sensor data streaming into floorplan in HomeAssistant

About Modding the Smart Plugs

Yeah, this is a bit weird. We have to… reflash… the smart plugs. To make them less smart. It’s fine. By the time you’re done you can probably professionally install this stuff for rich people making more than your current day job and charging way less than what the OnQ type systems cost. Sadly.

TODO probably separate video for this.

Most of the smartplugs also have energy usage reporting

TODO Show how to setup energy usage graphs on the breakers

The HA smart plug integration can already graph the usage data. How cool is that! It does need a little love to get those smart breakers working similarly.

I’d like to break this data out into a separate Graphana dashboard but not there yet.

Minding Energy Usage, comfort

Since I have a boiler (for extreme cold use only) and two heatpumps, things can get a bit weird. The boiler is great for bringing the house up to 50-60 degrees inexpensively. The recommended solution was this Honeywell system which supports controlling multiple devices. It was designed for a geothermal system that has a heatpump assist, as a kind of two-stage, but my boiler technician friend has used this for similar setups with boilers and, so far, it seems to work well. The boiler can do its work to bring the base temperature of the house up and either the circulator fan or fan+heatpump can move the air around or provide more fine-grained temperature control.

The curves for what system does what, when, I am still playing with. Part of the magic here depends on an outdoor temperature sensor (which is part of the honeywell system).

All that complication aside, the house can run leaner energy usage at night when I’m just in my bedroom and nowhere else. The house can run leaner energy usage when I’m not here.

It’s on my todo list to create an app on my phone that when I’m approaching my house it sets the “present” temperature preference immediately so the house can be the right temp by the time I arrive.

All this is accomplished without expensive purpose-built systems or things like Nest Thermostats – standard honeywell z-wave capable thermostats make all this possible.

Enter Z-wave, like sandman.

TODO

The Right z-wave USB stick is this one. Z-wave is another rabbit hole you can go down; however Lowe’s is selling a lot of “standard” stuff like Smoke Detectors that are Z-wave. The smoke detectors for the bedrooms are Z-wave, rather than alarm panel, just to have another layer of redundancy. They’re battery operated and fully self-contained. They can also detect carbon monoxide

How does the o2 monitoring work?

Long story. This house had a bunker with an HEGA filter. That’s not a typo. I repurposed it from that… It’s normally about a $500-$1000 item added on to your central hvac system and by far the most spendy thing on this list. It has a simple open/close header, and is designed to be plugged into the controller of the HVAC and the HVAC will run it continuously. Most of the time the air exchanger automatically captures energy from the air coming in, or leaving, so the hvac doesn’t really have to do a lot of extra work heating or cooling the outside air – the idea is to just let in more o2 and let the co2 leave naturally.

I have this serial o2 monitor TODO

it takes a while to come up with accurate numbers. A good o2 sensor can only work with known temperature, humidity and pressure variables, so this one takes about 15 minutes to get a good reading. Your co2 level should never be over 1000 indoors. I was surprised how often my “yawning fits” at my desk were due to co2 being over 1000 at my desk. Since being more aware of the co2 levels where I’m working, I feel it’s helped my productivity at least somewhat.

This is the model I got on amazon if you want to order a monitor. It’s pretty awesome.

TODO

Wiring this up to the Pi is pretty easy – USb2 interface and done. The HA integration is pretty crap at this point, working on that. Fortunately we can just let the exchanger do its thing.

TODO

Creating a Guest Landing Page

This one is pretty easy – the idea is a guest in your home joins the guest wifi and the first web page they see is one you made that has all the stuff they can do in your house: Your plex media server, name of chromecast/airplay devices, shared public network folders… whatever. This is just a matter of setting up your wifi access point plus a local web server. It’s got a pretty good wow-factor for your guests though. Especially once they click into plex and don’t have to log in because it’s a local IP Address.

TODO how to also set that up since the HA image makes that a PITA.

There are a lot of words here. But once you have clarity in your understanding of the moving pieces, you’ll see how clean and easy this really is.

What do people do for whole house audio that makes sense and is not super costly? The most expensive bits here are the air exchanger and the pex flow monitors. Everything else is $100 or less.

this is an early draft. Thoughts or comments welcome. probably be a whole series of videos! ryan’s video on software defined radio is already done

12 Likes

this thread is glorious

3 Likes

how about a DIY robot arm that can pickup clothes from a nearby hamper , put them into the washer, start it, put them in dryer when done and start it, and then text me when they are done drying?

1 Like

As convenient as that would be it’s robotics, not automation.

If you’re searching for some Security Ideas, this video is pretty good.

3 Likes

On the topic of Triggering things in Home Assistant based on whether you are home or not (you mentioned outdoor lights specifically). I’d recommend looking into Bayesian Sensor Type in HA.

Its fantastic for aggregating sensors in a meaningful way. Presence detection being especially useful. Say your phone likes to turn off wifi when its asleep for a long period of time. That won’t help much for detecting you are home. but pair that with some other things and it can be more useful. I always have a smartwatch on my wrist that announces itself with BLE. So you could tie that into some motion sensors and your phone sensors and then it can make pretty confident guesses on whether you are home or not. Or even in a specific location of your home. Maybe I forgot my phone at home. but If there’s motion sensors and my smartwatch is picked up by ble, then I probably just got home. Or vise versa. Maybe I forgot my watch at home, I don’t want the lights to activate anyways.

I’d also suggest this for individuals who have more than one person in their home to keep track of who is likely home or not. If there’s motion in the living room and the sensor on my bedroom door opened in the morning, It was probably me getting up, not my child. So go ahead and turn on the kettle so I can make my coffee in the morning. If (when my kids are teens) in the night, the window opens to my boy’s room and his phone suddenly connects to the network. It may have been him sneaking in late at night. Just turn on my lights and send my phone a notification instead of setting off the alarm. etc.

The possibilities are endless the more sensors you add to your system. And it better approximates what a real human would try and do to figure out certain triggers and events without needing AI/ML to figure things out.

There’s a great reddit thread I read that introduced me to this. I can post it here if you are interested in it.

1 Like

It must be morphic spheres. During the last couple of weeks i also thought about how i could make iot better. But i was thinking about shielding Hesung´s iot devices, not making the devices themselves good.

My Idea : modern houses have ethernet outlets in every room. Just double that (I am an electrcian so:$$$)and have the iot on a network on a seperate switch, which is configured to only listen to local IPs, and the user would VPN into that.
i know that might be a ghetto solution but its technicaly rather simple.
this would increase the security to the level of the VPN software and prevents your stuff from showing up on shodan .io.

With regard to the air quality / Co2 sensor…
I have a Pi4, SI7021 (temp / humidity), SGP30 (TVOC / estimatedCo2), and a simple Python script to read from the sensors.
I’m not sure if this is the right place for coding help, but I’m looking to upload or stream the measurements from the Pi to an Influxdb server (for visualization in Grafana).
The script to read the data from the sensors seems to work as expected; however, attempting to format the data to JSON and offload it to Influx seems to be failing. I’m not a Python expert, but when I try to print what is being formatted as JSON, it’s just blank.
I can post the script if it’d help, or relocate this post if that’s more appropriate. Thanks!

Yes, a hard wired isolated network is really the only reasonable way to use most “smart”/IoT devices. The problem is that so many of them refuse to work, or stop working, after they can’t check in for so long. If one avoids those though, and keeps the smarts local without cloud interraction, it can work out quite well. Most people want to have a bridge to some sort of cloud item to show things on their phone though, and once you do that, the giant hole is there.

I´ve recently gotten into the IOT rabbit hole.

I´ve ended up using a raspberry pi that runs https://www.zigbee2mqtt.io/ (all the other stuff runs on another machine)

Bought one of those elctrolama sticks (from the list of sticks they recommend). It comes without a working firmware, but is extremely easy to flash. Zigbee2Mqtt has it´s own firmware that you need to flash onto the stick to work with it.

It also has a setting that makes it create a seperate topic in mqtt specifically for home assistant. It describes the devices connected in a way home assistant understands. So when you connect it to your mqtt broker it will pick up the devices immidiately without any configuration.

For automating stuff I started using node-red. Curretly it´s bit unspectacular as I just got lights. Basically, I just use home assistant as a pretty interface for my phone currently.

Throw a VPN in the mix you got all that without a cloud.

For what it works with it´s pretty great (and there is a long list). Though there will probably be things that I´ll have to cherry rig together some other way. Like my AC. The only way I know how my AC is controllable is with an infrared remote. No idea how I´ll do that yet. But I wanna try.


I think ideally what Im doing is somewhat similar to a seperate switch. If ideally it where as secure as a physical wire. That raspberry is the seperate switch. Doing this over ethernet is a lot of spagetti in the wall to the point it´s not manageable. Just with lighting alone. I now have 1-2 lights in every room and 1-2 light switches in every room. Kind of both need to be “smart” because otherwise if you turn off the power to the light, the light is not only off it´s gone with no way to turn it back on other than flipping the power switch back on. Double seems not NEARLY enough.

This video is from 2015. Hope that things have changed a bit since then.

Most of the exploits he is talking about revolve around getting hold of the network key. I know in zigbee2mqtt you can deffinitely recreate the network key (where as he pointed out that with many commercially available zigbee gateways - or back then all he tested - you cannot do that and you have to throw out your zigbee gateway to get a new key). In zigbee2mqtt you can self generate one, type that in, or delete it and replace it with GENERATE and it will create one and replace GENERATE with the new key on next start. Not sure how secure the pairing process is these days. But there where ways to get hold of the key when you are pairing a new device to it. Some devices also appearently support remote reseting it (for your “convinience”). So you could reset that device and pair it to your own zigbee network remotely to get control of it.

I don’t remember if power monitoring at the breaker panel was mentioned in one of the vids. I would love more details about the options available to do some per-circuit monitoring.

I assume ring or clamp monitors would be the go-to here. Which ones are worth buying? DIY-ing?

What about power? Will these monitors need their own circuit inside the breaker panel to monitor the other circuits? (Yo dawg…) Is there a safe way to do this?

There is the " Kill-o-watt" I believe it’s called. Two clamps over the mains and it tells you your usage.