Thursday, August 10, 2017

(#28) That doggone total eclipse is gonna mess me up


It just dawned on me (pun soon to be intended) that the looming Total Eclipse is going to break one of my more useful web services for about an hour.  That web service is my "is it dark outside?" web service.


Sunrise, sunset...

In Home Automation, it's handy to know when the sun rises, when it sets and where we are at any given time between those two boundaries.  Here's my implementation in PHP:

<?php
 $aLatitude = htmlspecialchars($_GET["lat"]);
 $aLongitude = htmlspecialchars($_GET["lon"]);
 $aTimeZone = htmlspecialchars( (isset($_GET['tz'])) ? $_GET['tz'] : 'America/Denver' );

 //
 // uncomment one zenith setting: http://en.wikipedia.org/wiki/Twilight
 $zenith = 90+(50/60); // True sunrise/sunset
 //$zenith = 96;           // Civilian Twilight - Conventionally used to signify twilight
 //$zenith = 102;         // Nautical Twilight - the point at which the horizon stops being visible at sea.
 //$zenith = 108;         // Astronomical Twilight - the point when Sun stops being a source of any illumination.

 //
 // Good enough approximation at offset
 $tzOffset = $aLongitude * 24 / 360;

 //
 // But evidently, I need to deal with Daylight Saving Time myself
 if (date( 'I' )) {
    $tzOffset += 1;
 }


 $sunriseStr = date_sunrise( time( date_default_timezone_set( $aTimeZone )), SUNFUNCS_RET_STRING, floatval($aLatitude), floatval($aLongitude), $zenith, $tzOffset );
 $sunsetStr  = date_sunset(  time( date_default_timezone_set( $aTimeZone )), SUNFUNCS_RET_STRING, floatval($aLatitude), floatval($aLongitude), $zenith, $tzOffset );

 $isDark = FALSE;

 $date1 = DateTime::createFromFormat( 'H:i', date( 'H:i' ));
 $date2 = DateTime::createFromFormat('H:i', $sunriseStr );
 $date3 = DateTime::createFromFormat('H:i', $sunsetStr );

 if ($date1 < $date2 || $date1 > $date3)
 {
    $isDark = TRUE;
 }
 
 $result_json = array( 'sunrise' => $sunriseStr,
                       'sunset' => $sunsetStr, 
                       'currentDateTime' => $date1, 
                       'timezone' => $aTimeZone,
                       'inDST' => (date( 'I' ) ? '1' : '0'), 
                       'isDark' => ($isDark ? '1' : '0') );

 // headers to tell that result is JSON
 header('Content-type: application/json');

 // send the result now
 echo json_encode($result_json);
?>


It fits my needs. Pass in a Lat/Lon for the sunrise and sunset times at that location. Leave them off and it assumes you live near me.

It also returns the current date, time and timezone data for, you guessed it, me.




Invocation is pretty simple. Here's today's sunrise/sunset request for Seattle:

http://<yourwebserver:port>/<path>/sunrise.php?lat=47.6lon=-122.3

The result comes back as JSON:

 
   "sunrise":"05:48",
   "sunset":"20:21",
   "currentDateTime": 
      "date":"2017-08-10 09:29:00.000000",
      "timezone_type":3,
      "timezone":"America\/Denver"
   },
   "timezone":"America\/Pacfic",
   "inDST":"1",
   "isDark":"0"
}

Again - that currentDateTime object is for me.  Change it or remove it to fit your needs.  Once this webservice is in place, having other applications check to see whether it's dark outside or not is also pretty easy. A contrived python example:

import json
import urllib2

# My website needs authentication
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
use it instead of None.
top_level_url = "http://<yourserver>/<path>/sunrise.php?lat=40&lon=-105"
password_mgr.add_password(None, top_level_url, <usr>, <passwd>)

handler = urllib2.HTTPDigestAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)

data = urllib2.urlopen(top_level_url)
json_payload = json.load(data)
isdark = json_payload['isDark']

# do something
if isdark == "1":
   print "It is dark - turn on the outside lights"
else:
   print "It is light (not dark) - turn off the outside lights"





Wednesday, August 9, 2017

GOSUB 200 - Another Interruption - Daemons and Python


Plunking "instructions to myself" in a blog is useful, so here it comes again. This time it's about Python, Daemons and "systemd".

For code that I want to run all of the time and have it managed by the operating system there are a handful of approaches: write it as a daemon, write it as a executable and work up some cron scripts to keep it going, and now there's systemd.  Like it, revile it -- it looks like systemd is here to stay.

So, briefly, here's what I did to make my Python code managed by the systemd service.

It starts with a 'unit' file.  Bear in mind I have precious little information on what all of the options are, what the ones I've used actually do and the ramifications of the choices made.  

But here's what's working for me.

Step 1 - Create <useful-name>.service

$ more garagedoorcontroller.service
[Unit]
Description=Garage Door Controller Service
After=syslog.target

[Service]
Type=simple
User=myname
Group=mygroup
WorkingDirectory=/complete/path/GarageDoorController
ExecStart=/usr/bin/python /complete/path/GarageDoorController/GDController.py
StandardOutput=syslog
StandardError=syslog
Restart=always
RestartSec=60

[Install]
WantedBy=default.target

#
# After this the steps are:
# Create the unit file in /etc/systemd/system (the file name
# before .service will be the name you call to start/stop/restart the service)
# Set file permissions to 644
# Tell systemd that there is a new unit file: systemctl daemon-reload
# Start service: systemctl start <service>
# Check the service status: systemctl status <service>


Steps 2 and 3 - Copy it to the right spot


foo: $ sudo cp garagedoorcontroller.service /etc/systemd/system/.
foo: $ sudo chmod 644 /etc/systemd/system/garagedoorcontroller.service

Step 4 - Tell systemd that there's a new (or altered file)


foo: $ sudo systemctl daemon-reload


Step 5 and 6 - Start it, check the status


foo: $ sudo systemctl start garagedoorcontroller
foo: $ sudo systemctl status garagedoorcontroller

garagedoorcontroller.service - Garage Door Controller Service
   Loaded: loaded (/etc/systemd/system/garagedoorcontroller.service; disabled)
   Active: active (running) since Wed 2017-08-09 09:35:17 MDT; 18min ago
 Main PID: 12961 (python)
   CGroup: /system.slice/garagedoorcontroller.service
           └─12961 /usr/bin/python /complete/path/GarageDoorController/GDController.py

Aug 09 09:35:17 rpi4 systemd[1]: Started Garage Door Controller Service.



Tail /var/log/syslog if you have problems. The error messages are descriptive.



The BSD daemon, also called Beastie (a near homophone of the letters B-S-D pronounced slurred together), as drawn by John Lasseter. His widely known and popular take on the BSD mascot first showed up on a book cover in 1988. 

[ From the Wikipedia entry on BSD Daemon. ]

Monday, August 7, 2017

(#27) - Home Automation, Lights and Normal People


Normal

Here's how normal people turn on the lights in the kitchen:

Not Normal

And here's how people enamored with Home Automation turn on the lights in the kitchen:

  1. Fumble around in the dark, pull smartphone from pocket
  2. Swipe around until Home Automation App found, launch app
  3. Wait while Home Automation app starts
  4. Swipe around and find the Lighting Tab on the Home Automation app
  5. Swipe around and find the room labeled Kitchen
  6. Press the On button



And the industry wonders why adoption just won't take...

Still, that's far too simple and straight-forward for me, so continuing our TCP Connected Lighting Gateway programming from the last post, lets cover two of the four commands we use.

DeviceSendCommand


DeviceSendCommandLevel


To control individual bulbs, you use one of these two commands. The first turns a bulb on and off. The second command will set the dimming level.

#------------------------------------------------------------
def set_light_value(self, did, value=1, update=True):
    rc = self.my_gateway.send_device_command(did, value)
    if update:
        self.load_rooms_and_devices()
    return rc

#-----------------------------------------------------------
def set_light_level(self, did, level=100, update=True):
    rc = self.my_gateway.send_device_command_level(did,level)
    if update:
        self.load_rooms_and_devices()
    return rc



And then the HTTPS calls to the gateway are:

















The two strings, DeviceSendCommand and DeviceSendLevelCommand were shown back here.


The last two commands of the four affect an entire room and they're used in the same manner:

RoomSendCommand

RoomSendCommandLevel

#------------------------------------------------------------
def set_room_value(self, rid, value=1, update=True):
    rc = self.my_gateway.send_room_command(rid,value)
    if update:
        self.load_rooms_and_devices()
    return rc

#------------------------------------------------------------
def set_room_level(self, rid, level=50,update=True):
    rc = self.my_gateway.send_room_command_level(rid,level)
    if update:
        self.load_rooms_and_devices()
    return rc

No surprise here, the HTTPS calls to the gateway are:

















The "update" variable is used to potentially postpone a call to reload the room and bulb status values. That call, the RoomGetCarousel call is a bit 'thick' and I wanted to avoid calling it if I'm doing multiple bulb updates,

(#26) - Let there be light! 


Insufficient coffee to fuel any witticisms or snarky remarks, means I'll cut to the chase. The lighting controller in my house is from a company called TCP Connected. I've discussed the challenges I've had with their product back here.  

Bottom line: it works (it controls the lights well), the company has abandoned the product and I've invested enough in the product that I don't want to abandon it.



Standing on the Shoulders -- Again

Several others out there in 'net land must feel the same way, in that they've reverse engineered the protocol used and resurrected the product. Thanks to folks like this person here, we're wresting control of the lighting gateway from the manufacturer and making the thing useful again.  Thank you!


The Code Chase We're Cutting to (towards?)

If you promise not to make fun of my Python, I'll share the relevant snippets.


RoomGetCarousel 

The RoomGetCarousel command is the command sent to the Lighting Gateway that returns a lot of information in an XML payload. In this large XML stream, you'll find the name of the rooms, the names of the bulbs, and their state.

[ Nota Bene: You may be able to update the Gateway with new room and bulb information, but I've been doing that using the phone application when connected locally. ]


I have a TCPGatewayManager object who's responsibility is to send the https commands to the gateway
.  



The Gateway supports SSDP but mine's assigned a dedicated IP address so I don't bother with that discovery protocol.


Here's the call to the Gateway using HTTPS:




Parsing the XML payload slowly to discover rooms, bulbs and other neat things:

def load_rooms_and_devices(self):
    logging.debug('reloading rooms and devices from gateway')
    xmldata = self.my_gateway.get_response_room_get_carousel_command()
    root = ET.fromstring(xmldata)
    gwrcmd = root.find('gwrcmd')
    gcmd = gwrcmd.find('gcmd')
    gdata = gwrcmd.find('gdata')
    gip = gdata.find('gip')
    returnCode = gip.find('rc').text

    #
    # Iterate through the rooms and devices
    # Bear in mind that you must send the right XML to the
    # RoomGetCarousel command
    # to retreive all of these fields below.
    self.rooms = []
    self.lights = []
    self.light_names = []
    for a_room in gip.findall('room'):
        r_id = a_room.find('rid').text
        r_name = a_room.find('name').text
        r = {r_name: r_id}
        self.rooms.append(r)

        ## print 'Room ID: ', r_id, ' Name:[', r_name, ']'
        for a_device in a_room.findall('device'):
            a_did = a_device.find('did').text
            a_name = a_device.find('name').text

            a_state = a_device.find('state').text
            if a_state == '1':
               a_device_powered_on = True
            else:
                a_device_powered_on = False

            if a_device.find('offline') is not None:
                a_online = False
                a_level = "Unknown"
            else:
                a_online = True
                a_level = a_device.find('level').text
            ## print '   Name:[', a_name, ']  DID:', a_did, '  
            ## Powered On:', a_device_powered_on, 
            ## '  Connected To GW:', a_online, '  Level:', a_level

            a_bulb = {'name': a_name, 'did': a_did, 'level': a_level, 
                      'on': a_device_powered_on,
                      'connected': a_online, 'rname': r_name, 
                      'rid': r_id}
            self.lights.append(a_bulb)

            self.light_names.append({a_name: a_did})



I'm still playing with the Python structures that hold my return values. Right now I care about: rooms (their names and numeric IDs) and bulbs (their name, their Device ID [did], their state and their room).


Next post will cover how to turn the lights on, off and dim them.


Tuesday, June 20, 2017

(#25) My IoT Garage Door Closer


There's a big problem with leaving your garage door open at night.

And the problem is that the thieves never take the stuff you want them to take. 

Do they grab the box of outdated college text books from 1985? 
No. 

Do they grab the mostly-empty can of Oldham Tudor Earth Tone brown paint you bought in 1995 when that color was in vogue? 
No. 


How about the bicycle pump that still works, sort of? No. The really-broken-but-I-can-fix-it lawn chair? Nope.


Then there's a big problem with big problems (in my head) and that's that I tend to avoid the cost-effective and highly reliable solutions out there and pursue a DIY approach.

And there's a big problem with a dearth of clip-art about leaving your garage door open at night. So instead of a nice picture of a wide-open garage door, I'm using a picture of what I can only surmise is a puzzled burglar who came completely equipped with a big bag of break-in tools only to find a house, well lit with windows and doors wide open.



Admiral Gial Ackbar's warning seems appropriate.


I'm not naming names...


So, someone in our house tends to leave the garage door open. And that's a problem. We've already got the sensors installed to detect the open door. All we need to do is provide a means to close it.

The 'net has dozens of posts on how to build a garage door closer with a Raspberry Pi and a relay and I stole one approach from here. Thank you Andrew https://github.com/andrewshilliday/garage-door-controller

My Raspberry Pi 2B controller is wired like this:


5V power to the relay comes from an old, USB charger cable.

Python Relay Control

The python code to control the relay is simple

import RPi.GPIO as GPIO
from time import sleep


class Relay(object):

   def __init__(self):
       print 'Creating and initializing relay class'
       GPIO.setwarnings(False)
       GPIO.setmode(GPIO.BCM)

       # I've wired door 1 to relay 1 to GPIO pin 23
       # door 2 is relay 2 on GPIO pin 24
       # sleep 0.25 seconds between relay trigger low/high to simulate button press

       self.door1_pin_number = 23
       self.door2_pin_number = 24
       self.latch_time = 0.25

       # Set relay pins as output
       GPIO.setup(self.door1_pin_number, GPIO.OUT, initial=GPIO.HIGH)
       GPIO.setup(self.door2_pin_number, GPIO.OUT, initial=GPIO.HIGH)

   def trigger(self, door_id=1):
       if door_id == 1:
           pin_number = self.door1_pin_number
       else:
           pin_number = self.door2_pin_number

       print 'Relay Trigger invoked on pin number', pin_number
       GPIO.output(pin_number, GPIO.LOW)
       sleep(self.latch_time)
       GPIO.output(pin_number, GPIO.HIGH)

   def cleanup(self):
       print 'Cleaning up relay'
       GPIO.output(self.door1_pin_number, GPIO.HIGH)
       GPIO.output(self.door2_pin_number, GPIO.HIGH)
       GPIO.cleanup(self.door1_pin_number)
       GPIO.cleanup(self.door2_pin_number)


Passing MQTT Messages (AGAIN!)


My Home Automation approached is based on systems passing JSON messages over MQTT. That means it's time to define the JSON messages that'll control our controller.

The Command Packet:
{  
   "topic":"GDCTL/COMMAND",
   "datetime":"2017-06-20T20:09:49",
   "doorID":1,
   "command":"close"
}

Valid commands are “open”, “close” and “trigger”. The first two do what you'd expect, except if an open door receives and open command – it's ignored. Likewise, a close command to a closed door is ignored. I added the “trigger” command to just activate the relay regardless of the state.

Every five seconds or so, the Status Packet is published:
{  
   "topic":"GDCTL/STATUS",
   "datetime":"2017-06-20T20:09:49"
   "door":{  
      "door_id":1
      "state":"OPENED",
      "state_datetime":"2017-06-20T20:09:1",
      "last_command":"OPEN",
      "last_command_datetime":"2017-06-20T19:51:35",
   },
   "system":{  
      "uptime":"23:40:32.84",
      "cpu_temp":92.5,
      "ssid":"ESSID411",
      "camera_present":true
   },
}

The first datetime is the current date / time of packet creation. Valid door states are 'Unknown', 'Opened', 'Closed' and 'In-Motion'.

The system object simply sends back some information about the RPi that I'll find useful.


If you look at older blog posts you may notice that there are some changes afoot in my design. Over the years now of accumulated MQTT usage, I've concluded that I want the MQTT topic to be part of the payload. I'll spare you the arguments for/against I've had with myself. But, I want the topic to be part of the payload.

You can see in my older MQTT messages, the topic is outside the JSON payload. I've changed my mind on that too now – the entire MQTT packet payload is a JSON formatted messages and the topic is part of the message.