(#9) Complex : adjective kämˈpleks,kəmˈpleks,ˈkämˌpleks/

Our content's about to slow because we're catching up to where I'm still learning myself.

If you're dropping into this blog at this post, let me catch you up: I've asserted that the last 40 years of Home Automation has produced crap and I've take a stab at why I think so. I've told you that I have a couple of systems scattered around my house that broadcast events.

I've told you that it dawned on me that drawing conclusions from (a) the occurrence of an event or (b) the occurrence of a pattern of events is my next task.

CEP - Complex Event Processing
I first heard the term back around 2004, give or take, when I went out to Mountain View, CA to visit some of the engineers at TIBCO. Vivek stopped by for a few minutes and sort of asked if we had heard of this "really cool, new thing." And then he walked us through the basics of using events, specifically patterns of events, to draw conclusions.

If I recall correctly, he told us of a particular North American railroad company that was using it to drive costs out of their infrastructure.

My head jumped to the thought: all that track, all those sensors, all those cars and all that legacy infrastructure. 

If they're into this technology, it has to be approachable.

I won't go too much father into the science, for the most part because I don't know much of the science about this stuff. But I will say if you are interested, start off with two authors who are (or, in my opinion appear to be) the Fathers of CEP: David Luckham and Opher Etzion.

Life Without You

Think, just for a moment, how'd you'd code up something like this.  You'd code an MQTT subscriber, and listen for the first event.  When it came in you'd jump to a state where you'd listen for the second event.  When that one arrived, you'd jump to a state where you'd listen for the 3rd and final event.

Not too bad.

Until you realize that you need to switch things around.  Maybe swap the arrival of events 1 and 3.  Or you want to time-box the whole thing.  Event 3 must come no later than two minutes after event 1. Or maybe you want to add a 4th event.

Yuk! Pretty quickly it should dawn on you that hard-coding this will quickly become frustrating and inflexible.

Wrap it up Already!

Ask and ye shall receive. Here's the solution.

String  anEPLQuery = "SELECT * FROM PATTERN " +

  " [  every eventOne = HHBStatusEvent(macAddress='0000012467',deviceStatus='OPEN') " +

  " -> eventTwo   = HHBStatusEvent(macAddress='000000B357',deviceStatus='OPEN' ) " +

  " -> eventThree = HHBStatusEvent(macAddress='000000B357',deviceStatus='CLOSED' ) ].win:time( 5 minutes )";

Now all you need to know is abit more about the problem!

But this one Java String...

This one simple Java String...

Captures the essence of this requirement.

1) Subscribe to the event stream

2) When you see the following event pattern

2.1) An Open Event from Sensor 0000012467

2.2) Then an Open Event from Sensor 000000B357

2.3) And then a Closed Event from the same Sensor (000000B357)

2.4) And all three events have taken place within a 5 minute sliding window

3) Then do something

It's so smart, it's got to be using something like ESP (Extra Sensory Perception)!
But hold that ESP thought for a moment, I want to take a quick (one post) detour into ALARMS!

(#8) Introducing Homeminder! 

HomeMinder is a programmable electronic system that controls lights, appliances, heating, and cooling in your home through existing house wiring. Visuals on a TV set are used for system set up and operation. It is also possible to control devices by using a smart telephone.

HomeMinder is available in two versions--a free-standing unit that connects to any TV set (or monitor) and a unit built into a TV set. 

The main controller comes packaged with a remote transmitter, one lamp module, one appliance module, and appropriate cables. Also available are extra modules, a light switch module, thermostat controller, and remote control unit.

Is It Worth It?

HomeMinder retails for about $500 with an assortment of seven or eight modules. 

Wow!

Sounds pretty cool.  Not the $500 part. But it sounds like a slick Home Automation System.

I guess it is pretty cool. Or was. In 1984.

Homeminder was "GE Homeminder", a product of a joint venture between Pico/X10 and GE in 1984.

So, it died. 

It went nowhere.

Failure.

(You can read more about Pico/X10 and their products in this article from October 1999).

What's up with all of these failures?

Why have all of these products either shown modest acceptance or made large impact craters?

X10, Insteon, Z-Wave, UPB, Zigbee.

I guess these are as much protocol as product; it's not fair to call a protocol a failure. And there's little value in naming product names.  Folks would simply argue about whether they'd had wide spread acceptance or not.

Big Orange Retail Giants

Or Blue ones - Lowes in this case -- took a run at it last Christmas (2013) with Iris. Read more about Iris here. 

Was Iris a commercial success? Only Lowes knows. (Hey -- I like that rhyme!)  

But raise your mouse if you know someone who bought one!  

Last time I looked in Lowes, the display had moved from Front and Center to back by the extension cords.  And there was still plenty of product.

I won't ding Iris. 

I have no idea what it can and cannot do.

I'll tell you what I didn't like about it.

• It was expensive - and subscription based. $10/month.
• It was closed and proprietary.
• A motion sensor was $25.
• A light switch $35.
• Their thermostat is $100.  I just bought a Nest.

Starter kits were $180 to $240.

I paid $25 for my Eaton Home Heartbeat starter kit.

What if?

What if I buy one and it sucks?  I'm out, what $200? $300?  More?

Will it work with my Nest?  I doubt it.

What if Iris decides to bail on the whole thing, like Eaton did?  What happens to my investment? Does it still work?  When iOS 12 comes out, will their app written for iOS 6 still work?

What if I don't like the way it does something? Can I change it?  Probably not.

Is it open? Hackable? Tinker-able?  Probably not.

These are questions that never, ever cross the mind of someone named Felix in Marketing.

Sorry - he''s just not wired to think these thoughts.

Those neurons don't exist in Felix's brain.

But here's the schnitz Felix - your Home Automation System -- the only ones interested are us engineers.

And all of those "What If's" that you didn't think, we've thunk.

And we walk on by.

Apple

The wild card.  If anyone can pull off a Home Automation System that's closed, proprietary, expensive and limited - it's Apple.  I bet they sell a gazillion of 'em.

Damn Fanbois.

Look at Phillips Hue!  $200 for three colored light bulbs and an iPhone app. I mean, they didn't even toss in a mirrored disco ball! Criminy!

Damn Fanbois.

I guess Life is Just Complex!

(#7) Nerd Out Time 

Let's pause for a minute and talk about some nerd specific topics. I've covered off a little bit on the hardware and sensors involved. And I've mentioned that they broadcast events. Let's dig a bit deeper into the nerdosity behind the event mechanism.

Mosquitto

I have zero desire to talk about the Internet of Things here, other than to say I believe it's coming soon and will be big.  Really big. Al-Gore-Internet big.

One of the technologies collecting interest in this space is the publish/subscribe message broker called MQTT.   (Go nuts, read more about it here.)

There are others. I've used a few of them.  But I've decided to invest some energy into using MQTT.  Why?  Because, that's why.  I'm using two implementations of MQTT. My broker, 'C' and Python applications are using the Mosquitto code.

It's really cool and has been working really well.  

On the Java side, I've been using the Paho MQTT Libraries.   Also cool, also working well.


Picture Time!
MQTT is pretty simple.  Run the broker code, someplace on your network. Mine's on a server, in a closet.  But you already knew about the closet part.

The publishers are applications on the network.

My Eaton Home Heartbeat controller code is one publisher.

My LaCrosse Technology WS-2308 AL Weather Station is another publisher.

Let's put off the subscriber for a paragraph or three and dig into those messages.

Events are Represented by MQTT Messages

That's right.  Events that occur in my house end up as published MQTT Messages. When the front door opens, this message is broadcasted:

HHB/STATUS | 2014-05-30 14:10:07 -0600 | 03 | OPEN-CLOSE SENSOR | Front Door | OPEN | 18000 | ALARM ON OPEN | NO ALARM ON CLOSE | DO NOT CALL ON OPEN | DO NOT CALL ON CLOSE | ONLINE | BATTERY OK | CLEARED | 0000010228 |

It's long, but let's bust it up and look at the parts of the message. The first thing to notice is that I use a vertical bar '|' to separate the fields. It makes parsing the string easier.

HHB/STATUS |                The MQTT Topic!
2014-05-30 14:10:07 -0600 | The date / time of the event
03 |                        The sensor type (Open / Close are type '3')
OPEN-CLOSE SENSOR |         The sensor type in English
Front Door |                Where the sensor is
OPEN |                      Open or Closed
10 |                        Number of seconds it's been open
ALARM ON OPEN |             Some Home Heartbeat Specific Stuff
NO ALARM ON CLOSE |         Some more
DO NOT CALL ON OPEN |       And some more
DO NOT CALL ON CLOSE |      And even some more
ONLINE |                    Online of Offline if the sensor is alive
BATTERY OK |                Battery state of the sensor
CLEARED |                   Alarm cleared or 'triggered'
0000010228 |                Unique MAC Address of the sensor

Could it get any more nerdy?

Yes, just a bit more.

Recall I'm looking for three events, in order.  These are the three events, as MQTT messages that I'm waiting for (some fields removed to save space!)

HHB/STATUS | 14:10:01  | 03 | OPEN-CLOSE SENSOR | Door to the Garage | OPEN |

HHB/STATUS | 14:11:01  | 23 | TILT SENSOR | Garage Door | OPEN |

HHB/STATUS | 14:13:01  | 23 | TILT SENSOR | Garage Door | CLOSED |

How hard can this be?
In the onslaught of network events, just write some code that looks for these three events in this sequence.

Said another way:
- write some easily maintainable modifiable code, since I may change my mind about the rules
- that subscribes to all these events
- events coming across as MQTT messages
- hammering the network at about four messages per second

And code that can recognize this pattern of events: open, open close.


This pattern of events.
Recognize this pattern of events.
After I code up this one I'll probably have other patterns to watch for.
Patterns.
Gang of Four - no - wrong pattern.

Patterns of events.
Why does that ring a bell?



Code that watches an event stream for patterns.
And reacts accordingly.


Ding!  This is why Patterns of Events rings a bell!

But first, a commercial for a slick Home Automation product that's sure to be a hit!.

(#6) Eureka! 

And no one has to jump, naked out of a bathtub.   Time for another recap: Home Automation has been around, at least, since 1975 and all of the other systems sucked.

Why?  Well for several reasons.  But here's a new one -- they're not as flexible as I'd like them to be. Recall that I've simply asked a (rhetorical) question:



Why can't my Home Automation know when someone has left the house, and turn the thermostat down in the winter?




I've got sensors, I've got a programmable thermostat.  All I need is a way to know when someone's left. Thinking about it, there are probably many ways to figure out when someone's gone:

• The Front Door opens and closes
• Their phone's bluetooth ID disappears from some bluetooth sensor

Yeahbut - the front door opens and closes when the neighbor comes for a visit. That won't work.  The bluetooth thing would be fun, but the range isn't that great.  You'd be out of range if you walked into the study.

Wait!
What about...?

The Morning Routine
We've already seen that around 6:30, on a school morning, I'm taking a shower. Around 7:30 something else happens that's very predictable, very repeatable:

1. The Back Door Opens
2. The Garage Door Goes Up
3. And about two minutes later The Garage Door Comes Down

Think about it.

That's an (almost) foolproof way of inferring that someone has left!  Open the door, raise the garage door, get in the car, drive off, close the garage door.

All I need to do is listen for these events in this order.

If I see these three events, in this order, then I can infer that someone has left!

Oooooooooooooooooooooooooooooooooo.

This should be fun.

Time to get our Nerd On!

(#5) I don't care - I want one!
The Nest thermostat, that is.


Let's recap on some things.  I started this off with a history lesson. And I said that other folks have been taking a run at Home Automation for 40 years. And they've all failed (or had modest success).

What was good about the 1975 Home Automation Technology X10?

• It was cheap.
• It worked, mostly.  Usually.  Mostly.
• If the protocol wasn't public, it was hackable.  It wasn't intentionally secured, obfuscated or encrypted.

So let's revisit them one at a time.

Money
Is what I have cheap?  Yeah. Relatively speaking.  The weather station was $90. The Eaton Home Heartbeat Starter Kit was about $20.  Sensors are $10 give or take.  The Raspberry Pi's are $35 each. I haven't mentioned another sensor, but I also have some temperature sensors that are under $10 each.

I recognize "affordable" is relative, but so far, I'd say what I have is affordable.

Reliable
Yes.  So far, the systems I have have been very reliable. Much more so than an X10 based system. 


Open
Yes.  While I favor items with an open, public, sanctioned and supported API, then next best thing are devices that have enough popularity to encourage other engineers reverse engineering their protocols.


And that brings us back to the Subject - the Nest Thermostat.


The Nest Thermostat
Definitely not affordable by my definition. $250 about.  That's a lot of money for a darned thermostat.

I'd been hemming and hawing for months.  Looking for more affordable alternatives. These guys, the Radio Thermostat company, were the leading candidate. Their devices get great reviews and they have an Open, Supported API!

They're perfect!

Stuff Happens
Then one very cold day, our Honeywell thermostat dies.  Did I mention it was a very cold, winter day?

So I need to get a replacement and I need to find one now.  And that's how I ended up with a Nest thermostat. 

An Aside
Ok - the damned thing is awesome.  Fricking amazing.  From installation to operation to the portal - 5 stars.   Hey Nest - you guys suck.  No one should be that good. You nailed the complete user experience.  I hate you.  You make the rest of us look bad.


Promises Promises
Late in 2013, Nest said they were working in a public API and shooting for a first quarter release date.  Well, it's almost June and there's no public API.

But no worries - just like all of the other devices in my house, some resourceful nerd took the challenge to heart and figured it out.

As of this writing it looks like there's a PHP solution, a Python one and a Java based API. I picked, JNest, The Java based way to talk to your Nest Thermostat.



So...?
So, with a little Java code, the Nest Theromstat was added to the Home Automation ecosystem. It pumps out events with the current temperature, the current humidity. It also tells me whether the furnace is on or not, if the AC is running, if it thinks the house is unoccupied.

It's a slick little sensor:

But wait, there's more!

The Nest thermostat also accepts commands.

Remember the Use Case I shared?  How come she always forgets to turn down the thermostat when she leaves the house?



We're getting there.
I've got sensors that'll help me figure out when the house is unoccupied.
And I've got a thermostat that I can control, programatically.

I think I'm about to have a Eureka Moment!

(#4) You Don't Need a Weatherman
There's another hunk of hardware in my house that contributes to the Slightly-Less-Dumb-House ecosystem: a weather station.

According to my unscientific, unpublished studies, most nerds exhibit an affinity for music and an interest in weather.

To feed that interest, yet acknowledge that there's little real value in the endeavor, I bought an weather station for about $90 USD.


LaCrosse - A Game, A City and

The maker of my inexpensive and accuracy impaired weather station is LaCrosse Technology. I picked the WS-2308 AL model for a handful of reasons.  First it was cheap, and second it came with the ability to communicate with a computer.

Open Source to the Rescue
The communication protocol between the weather station and the computer is closed and proprietary.  I think LaCrosse will sell you the specifications.  I'm not sure.  I'm sure I don't care.

As with my Eaton Home Heartbeat system, someone's invested the time to reverse engineer the protocol.  They've documented it and even written some software, freely available.

If you're curious, and if you happen to have one of these models (also discontinued) then you're welcome to browse Kenneth Lavrsen's Wiki Pages.


Another Closet, Another Pi and more C
Like the Home Heartbeat System, the weather station is in a closet, connected to another Raspberry Pi and has some of my 'C' code that uses the Open WS2300 protocol to query the weather station about current conditions.

The current conditions are broadcast to the network (more on this later) and find their way to a MySQL database on another server (yes, in a closet, but no - not a Raspberry Pi).

I've had this setup for several years. The weather station predates the Home Heartbeat System and the current conditions are viewable on a simple PHP based webpage.

While semi-interesting, it turns out that the weather station is just another set of sensors that can contribute to the Home Automation ecosystem.

Is there anybody out there?

I don't want to jump too far ahead, but a common use case I'm bumping into, with Home Automation, is trying to infer when the house is empty.  Sure -- plaster motion sensors everywhere. Toss in a lot of creepy cameras.  That would do it. But it would be creepy.

But take a look at this graph. This is from the weather station's indoor sensors. It shows the interior temperature and humidity:

Do you see that humidity spike about 6:30am?  That's me jumping in the shower. (Well, not literally jumping up and down while inside the shower.  I'm using an idiom. It means I've started the hot water running.)

And we've found another way of inferring that someone is home.

If we see a second floor, interior, humidity spike, between the hours of 6 and 7 am, Monday through Friday, then someone's home.  Me.

Interesting.

I didn't expect that.

Based on this event "a sudden rise in the humidity from the sensor on the second floor" we can assume the house is occupied.

We take the output of one event and make an assumption.

Come to think of it - I have another sensor, on the first floor, that measures temperature and humidity too.

(#3) Shout It, Shout It, Shout It Out Loud
By far and away, the fondest memories of my career, so far, are all rooted in the people I've met. I've had the opportunity to get to know some of the smartest, most thoughtful engineers on our planet.

For this post, one person, in particular, comes to mind.  In the mid 1990s, I was contracting at a large Telecommunication company headquartered in New Jersey.  One of their incumbent chief architects and I were butting heads over an inter-application protocol. I was advocating CORBA and he wasn't.

The circumstances aren't important.  What is important is, this person was right. And I was wrong. Gary Zidd, if you're reading this, please recall I bet you a nice dinner that CORBA would rule the world.  Pick the restaurant, order a nice dinner for you and the wife, and send me the bill.


What Gary was advocating was -- messaging. 
And he was so very, very, very right!


Are you Listening?
Mr. McGuire: I want to say one word to you. Just one word.
Benjamin: Yes, sir.
Mr. McGuire: Are you listening?
Benjamin: Yes, I am.


I'll save the Soapbox-ing for another post, but, dear reader - the future is in asynchronous messaging. The only path forward for us, mired in this muck we've created is asynchronous messaging.

In a half-hearted attempt to stay on the Home Automation topic, let's continue the discussion on a subclass of those asynchronous messages: events.


Academics To The Obfuscation
Leave it to the Ph.D.'s to take something as well-understood and intuitive as an "event" and complicate it.  Suffice to say, there's some interesting (yawn!) discourse on what constitutes an event.

For this post, please accept the following:

• an event is something that will happen, is happening or has happened
• and an event is also the message that captures such

For Example
Opening a door is an event. 

 And so is the message "The door has been opened!".   Ignore tense, ignore grammar. Note the academicians seem to prefer the past tense: an event is something that has happened.  But, pragmatically, it makes little difference.



Here's another example of an event: "The Inside humidity has exceeded 65%".  


And some more event examples:
"Someone just called from 303-555-1212"
"The motion sensor has gone quiet - nothing is moving"
"The water-leak sensor by the water heater is now reporting it's wet!"

Some non-Home Automation event examples:
"The 30 day moving average price for NetFlix stock has passed $400"
"The odometer on you car has crossed the next service interval threshold"
"The sun just set"
"The JVM heap space is 95% exhausted"


It's Just Thousand Island Dressing!
One of the ingredients in our Home Automation Secret Sauce is - an event.  Well, events, plural actually.  I'll share part of the Home Automation Secret Sauce Special Recipe with you:

Successful Home Automation Software:

• Is event based, is event driven
• Broadcasts, shares, these events with every other device on the network

Apple - are you listening?  I know you'll do the first and not do the second.  You'll create and consume events from your Home Automation ecosystem. But you won't share them.  You won't make them available to the rest of the world.

No.
They'll be locked up and sequestered inside your hub, your box, your controller.

That's myopic, at best.  Stupid, at worst.

Spray and Pray
My hacked-up cobbled together Home Automation fills my network with events. After watching it run for about a year, I think this is the path forward.

Sniff my network for these event messages (MQTT packets) and this is what you'll see:

It's unreadable, in this post, for a reason.

We'll cover off the event content later.

The point is - those events are being shared.
Those events are broadcasted.
Those events are visible to any other device, or application, on my network.


They're available to be consumed, interpreted, acted-upon.
In ways I haven't thought of yet.


The Answer to "Why Can't I..."
In an earlier blog post, I rhetorically asked "Why can't I turn the thermostat down when I know someone has left the house?"

The hard part becomes knowing when some has left, when someone isn't home.

I stumbled across something interesting the other day - an odd way to tell when someone is home.

(#2) The Heart of the Matter
At the heart of my home's brain (pun intended) is an Eaton Home Heartbeat System (second pun unintended).  The, now discontinued, Home Heartbeat System was introduced to the market around 2007 or so.

The product line consisted of a controller, a key FOB, sensors, a gateway, a range extender and access to an online portal.  In a nutshell, you placed the sensors around the house, plugged a phone line into the controller and then logged into Eaton's online portal.

As sensors triggered, the system could blink your key FOB or call/text you the appropriate alerts. 

Evidently, it didn't catch on. In 2011, Eaton announced that the line would be discontinued and online capabilities, including the calling/texting, would be turned off.   Consequently product flooded eBay and other outlet channels at fire-sale prices.

The Power of the Crowd
Smarter and more resourceful engineers picked up the discontinued systems and started to reverse engineer the protocols.  Two engineers were quite prolific in posting their findings.  I stumbled across Steve Davidson's blog posts first.  And then found this person's blog posts on Hacking the Home Heartbeat.

Their success, and the ridiculously low price on the Eaton Home Heartbeat Starter package, convinced me to give it a try.

The HHB Controller
In my house, the Home Heartbeat (HHB) controller is connected to a Raspberry Pi and stashed on a high shelf, in the coat closet on the first floor.  From here, the controller has good connectivity to these sensors:

• Door Open/Close sensors on all doors
• Garage Door Tilt/No Tilt sensors on the garage doors
• Water Leak Sensors (strategically placed; one by the water heater)
• Motion Sensors
• Electric Power Sensors

Armed with the protocol information on the two blogs, I wrote my own controller code (in 'C') to talk to the controller.

OK - Now What?
The story so far:

• I have a cheap system with wireless sensors
• I've Stood On The Shoulders of Giants and leveraged the work of others to make it work for me
• It's reliable

Contrast this with the X10 critiques from my first post and I think we're already ahead of the game.

But, what should I do with the sensor information flowing into my code from the controller? What do I want to do when a door opens?  What do I want to do when the motion sensor detects motion?

Thinking Bigger -- Why?

What do I want to do when the door opens?  Nothing really. The door opens and closes all of the time.  Maybe that contributed to the demise of the product.  Do I really want my key FOB to light up every time the door opens?  No.

Why...?

Why can't?

Why can't I have it be smarter?

Why can't I have the thermostat turned down, in the winter, when the house is unoccupied? Why can't the system text me when a door has been left open for more than a minute?

Why can't the system let me know when the garage door has been left open and it's dark outside?  I don't care if the garage door is left open Saturdays at 9:00am.  But I do care if it's been left open all night!

Finding a way to answer these questions became a quest.
But we need to take a small detour and talk about Event Driven architecture.