Wednesday, June 13, 2018

(#30) Solar Charge Controller Communication

The last post covered off on the hardware and computer connection for creating a communication link with the EPSolar LandStar LS1024B controller.

Here we'll briefly cover off on the basic software approach to communicating with the controller.

As mentioned last time, the communication protocol used by the controller is an archaic one called Modbus.

With our USB/RS485 Adapter connected and plugged into a Linux machine we can start coding up the software.

There are Python libraries for modbus, but I chose to use a 'C' library for initial exploration.  I used libmodbus V3.1.4 from on Ubuntu 16.04 LTS.

Modbus and Libmodbus

Fortunately the API exposed by libmodbus is small; there's not a lot to master. And the documentation is good. Unfortunately, the terminology is modbus specific and I've already taken the stance that I have no desire to master an interface protocol for industrial devices from the late 70's.

So rather than dig into why a register isn't a coil and why there are three or four ways to actually read data, it was easier to just throw some code and it and see what works.

The C code

Right off the bat, include the right header files:

#include <errno.h>
#include <modbus/modbus.h>

#define LANDSTAR_1024B_ID       0x01

Then try to open the port to the controller. The settings are 15200 Baud, 8N1. My USB/RS485 adapter shows up as /dev/ttyUSB0:

int main (int argc, char* argv[]) 
    modbus_t    *ctx;
    puts( "Opening ttyUSB0, 115200 8N1" );
    ctx = modbus_new_rtu( "/dev/ttyUSB0", 115200, 'N', 8, 1 );
    if (ctx == NULL) {
        fprintf( stderr, "Error opening\n" );
        return -1;

    printf( "Setting slave ID to %X\n", LANDSTAR_1024B_ID );

    modbus_set_slave( ctx, LANDSTAR_1024B_ID );

If the port opens successfully, then try to connect:

    puts( "Connecting" );
    if (modbus_connect( ctx ) == -1) {
        fprintf( stderr, "Connection failed: %s\n", 
                 modbus_strerror( errno ) );
        modbus_free( ctx );
        return -1;


Once the port is open, you need to consult the vendor documentation to see what 'modbus registers' hold the data we're interested in. 

Here's a snippet that shows how to pull what EPSolar calls the "real time" data:

void    getRealTimeData (modbus_t *ctx)
    int         registerAddress = 0x3100;

    int         numBytes = 0x13;  // 0x14 and up gives 
                                  // 'illegal data address'
    uint16_t    buffer[ 32 ];

    // zero everything out to start!    
    memset( buffer, '\0', sizeof buffer );
    // call 'read-input-registers', other read calls error
    if (modbus_read_input_registers( ctx, 
            registerAddress, numBytes, buffer ) == -1) {
        fprintf(stderr, "Read failed: %s\n", 
                modbus_strerror( errno ));
    // ---------------------------------------------
    //  Photo Voltaic Values - Volts, Amps and Watts
    float pvArrayVoltage =  ((float) buffer[ 0x00 ]) / 100.0;
    float pvArrayCurrent =  ((float) buffer[ 0x01 ]) / 100.0;
    // Assemble the Power (watts) value from two words
    long    temp = buffer[ 0x03 ] << 16;
    temp |= buffer[ 0x02 ];
    float pvArrayPower   =  (float) temp / 100.0;

A couple of notes...
There are a handful of 'read' calls in the modbus library. Again, rather that learn modbus details, I just whacked at the code until one of the calls worked.

But the 'read-input-registers' call did not always work. More on this later.

Note the 'register-address' variable that's set to 0x3100. Again, this value comes from the vendor documentation.  And the amount of data read, numBytes, can vary from 1 to MAX (where MAX seems to depend). 

This number of bytes values sometimes fell short of what the documentation said I could expect. For example, there are three additional fields documented:

float batterySOC = ((float) buffer[ 0x1A ]) / 100.0;
float remoteBatteryTemp = ((float) buffer[ 0x1B ]) / 100.0;

float systemRatedVoltage = ((float) buffer[ 0x1D ]) / 100.0;

But it didn't work for my controller - setting numBytes higher that 0x13 gave an error.

I mentioned that 'read-input-registers' didn't always work. For the Settings stored at 0x9000, I had to use the 'read-registers' call.  Looking at little into the Modbus specifics, I can see that one call uses Modbus function call 0x04 and the other uses 0x03.   It's most likely because the settings at 0x9000 are both read and write. 

At some point in the future, I'll try the function write-registers (modbus function 0x10) to see if I can update the settings. 

(#29) And now for something completely different

Tinkering on the Home Automation front has slowed. I've got a number of projects in various states of completion: IKEA Tradfri lighting, ODB2 and CANBUS monitoring, GPS work, etc.  And Solar.  

Since we bought the motorhome, my interest in solar charging of 12V systems has been growing.  A few months ago, I purchased a 100W solar panel, and a simple 10A solar charge controller to get started.

While this approach taught me a few things about solar charging systems at a low entry cost, the charge controller was dumb.  More sophisticated charge controllers, with communication capabilities are out there.

Given my interest in IoT, a smarter controller was a necessity!

The Solar Charge Controller

Beijing Epsolar Technology Co is a popular manufacturer of affordable solar charging products. Their LandStar series of PWM Charge Controllers met my needs of being affordable and having a communication interface. 

Twenty dollars and three weeks later, the LS1024B PWM Solar Charge Controller from EPSolar arrived.

Other models from this vendor used a serial interface. This one uses a protocol called "modbus".  I knew nothing of this protocol before the controller arrived and I still know precious little. And I have little desire to master an archaic protocol, so take this next section with a huge 'grain of salt'.

Apparently you can use the Modbus protocol over a variety of interface specifications, but the LandStar uses Modbus RTU over RS485.  While I have no idea what that really means, I was able to Google up a number of matches to get something cobbled up that works.

USB / RS485 Adapter

The first purchase was a USB/Modbus dongle.  There are dozens available on eBay, Aliexpress, etc. I spent $3.50 for one that looks like this and when plugged into Linux, it registers as "ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter":

Plug the adapter into your Linux machine, check 'dmesg' to ensure the device is recognized.  And you should have a "/dev/ttyUSBx" device appear.

[ ] usb 1-2: new full-speed USB device number 3 using ohci-pci
[ ] usb 1-2: New USB device found, idVendor=1a86, idProduct=7523
[ ] usb 1-2: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ ] usb 1-2: Product: USB2.0-Serial
[ ] usbcore: registered new interface driver usbserial
[ ] usbcore: registered new interface driver usbserial_generic
[ ] usbserial: USB Serial support registered for generic
[ ] usbcore: registered new interface driver ch341
[ ] usbserial: USB Serial support registered for ch341-uart
[ ] ch341 1-2:1.0: ch341-uart converter detected

[ ] usb 1-2: ch341-uart converter now attached to ttyUSB0

Wiring it all up

Wiring of the device is simple.  Grab any old Ethernet cable (the connection on the LandStar controller is RJ45.)  Cut one of the RJ45s from the cable, strip the wires.  Find the wires that connect to pins 3 and 5 - and trim the insulation.

Wire 5 goes the the connection marked "A" on the adapter. Wire 3 goes to the connection marked "B".  Use a multimeter to ensure continuity and a good connection.  Then plug the adapter back into the Linux machine.

Plug the other end of the Ethernet cable into the COM port on the controller.  Connect the 12V battery to the charge controller and the LandStart will power up.

[ Ignore the DC-DC buck converter in the photo as I use this setup to power a Raspberry Pi. ]

With the hardware wired and connected, we can move on to the software. Which is discussed in the next post!

Thursday, August 10, 2017

(#28) That doggone total eclipse is gonna mess me up

It just dawned on me (pun soon to be intended) that the looming Total Eclipse is going to break one of my more useful web services for about an hour.  That web service is my "is it dark outside?" web service.

Sunrise, sunset...

In Home Automation, it's handy to know when the sun rises, when it sets and where we are at any given time between those two boundaries.  Here's my implementation in PHP:

 $aLatitude = htmlspecialchars($_GET["lat"]);
 $aLongitude = htmlspecialchars($_GET["lon"]);
 $aTimeZone = htmlspecialchars( (isset($_GET['tz'])) ? $_GET['tz'] : 'America/Denver' );

 // uncomment one zenith setting:
 $zenith = 90+(50/60); // True sunrise/sunset
 //$zenith = 96;           // Civilian Twilight - Conventionally used to signify twilight
 //$zenith = 102;         // Nautical Twilight - the point at which the horizon stops being visible at sea.
 //$zenith = 108;         // Astronomical Twilight - the point when Sun stops being a source of any illumination.

 // Good enough approximation at offset
 $tzOffset = $aLongitude * 24 / 360;

 // But evidently, I need to deal with Daylight Saving Time myself
 if (date( 'I' )) {
    $tzOffset += 1;

 $sunriseStr = date_sunrise( time( date_default_timezone_set( $aTimeZone )), SUNFUNCS_RET_STRING, floatval($aLatitude), floatval($aLongitude), $zenith, $tzOffset );
 $sunsetStr  = date_sunset(  time( date_default_timezone_set( $aTimeZone )), SUNFUNCS_RET_STRING, floatval($aLatitude), floatval($aLongitude), $zenith, $tzOffset );

 $isDark = FALSE;

 $date1 = DateTime::createFromFormat( 'H:i', date( 'H:i' ));
 $date2 = DateTime::createFromFormat('H:i', $sunriseStr );
 $date3 = DateTime::createFromFormat('H:i', $sunsetStr );

 if ($date1 < $date2 || $date1 > $date3)
    $isDark = TRUE;
 $result_json = array( 'sunrise' => $sunriseStr,
                       'sunset' => $sunsetStr, 
                       'currentDateTime' => $date1, 
                       'timezone' => $aTimeZone,
                       'inDST' => (date( 'I' ) ? '1' : '0'), 
                       'isDark' => ($isDark ? '1' : '0') );

 // headers to tell that result is JSON
 header('Content-type: application/json');

 // send the result now
 echo json_encode($result_json);

It fits my needs. Pass in a Lat/Lon for the sunrise and sunset times at that location. Leave them off and it assumes you live near me.

It also returns the current date, time and timezone data for, you guessed it, me.

Invocation is pretty simple. Here's today's sunrise/sunset request for Seattle:


The result comes back as JSON:

      "date":"2017-08-10 09:29:00.000000",

Again - that currentDateTime object is for me.  Change it or remove it to fit your needs.  Once this webservice is in place, having other applications check to see whether it's dark outside or not is also pretty easy. A contrived python example:

import json
import urllib2

# My website needs authentication
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
use it instead of None.
top_level_url = "http://<yourserver>/<path>/sunrise.php?lat=40&lon=-105"
password_mgr.add_password(None, top_level_url, <usr>, <passwd>)

handler = urllib2.HTTPDigestAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)

data = urllib2.urlopen(top_level_url)
json_payload = json.load(data)
isdark = json_payload['isDark']

# do something
if isdark == "1":
   print "It is dark - turn on the outside lights"
   print "It is light (not dark) - turn off the outside lights"

Wednesday, August 9, 2017

GOSUB 200 - Another Interruption - Daemons and Python

Plunking "instructions to myself" in a blog is useful, so here it comes again. This time it's about Python, Daemons and "systemd".

For code that I want to run all of the time and have it managed by the operating system there are a handful of approaches: write it as a daemon, write it as a executable and work up some cron scripts to keep it going, and now there's systemd.  Like it, revile it -- it looks like systemd is here to stay.

So, briefly, here's what I did to make my Python code managed by the systemd service.

It starts with a 'unit' file.  Bear in mind I have precious little information on what all of the options are, what the ones I've used actually do and the ramifications of the choices made.  

But here's what's working for me.

Step 1 - Create <useful-name>.service

$ more garagedoorcontroller.service
Description=Garage Door Controller Service

ExecStart=/usr/bin/python /complete/path/GarageDoorController/


# After this the steps are:
# Create the unit file in /etc/systemd/system (the file name
# before .service will be the name you call to start/stop/restart the service)
# Set file permissions to 644
# Tell systemd that there is a new unit file: systemctl daemon-reload
# Start service: systemctl start <service>
# Check the service status: systemctl status <service>

Steps 2 and 3 - Copy it to the right spot

foo: $ sudo cp garagedoorcontroller.service /etc/systemd/system/.
foo: $ sudo chmod 644 /etc/systemd/system/garagedoorcontroller.service

Step 4 - Tell systemd that there's a new (or altered file)

foo: $ sudo systemctl daemon-reload

Step 5 and 6 - Start it, check the status

foo: $ sudo systemctl start garagedoorcontroller
foo: $ sudo systemctl status garagedoorcontroller

garagedoorcontroller.service - Garage Door Controller Service
   Loaded: loaded (/etc/systemd/system/garagedoorcontroller.service; disabled)
   Active: active (running) since Wed 2017-08-09 09:35:17 MDT; 18min ago
 Main PID: 12961 (python)
   CGroup: /system.slice/garagedoorcontroller.service
           └─12961 /usr/bin/python /complete/path/GarageDoorController/

Aug 09 09:35:17 rpi4 systemd[1]: Started Garage Door Controller Service.

Tail /var/log/syslog if you have problems. The error messages are descriptive.

The BSD daemon, also called Beastie (a near homophone of the letters B-S-D pronounced slurred together), as drawn by John Lasseter. His widely known and popular take on the BSD mascot first showed up on a book cover in 1988. 

[ From the Wikipedia entry on BSD Daemon. ]

Monday, August 7, 2017

(#27) - Home Automation, Lights and Normal People


Here's how normal people turn on the lights in the kitchen:

Not Normal

And here's how people enamored with Home Automation turn on the lights in the kitchen:

  1. Fumble around in the dark, pull smartphone from pocket
  2. Swipe around until Home Automation App found, launch app
  3. Wait while Home Automation app starts
  4. Swipe around and find the Lighting Tab on the Home Automation app
  5. Swipe around and find the room labeled Kitchen
  6. Press the On button

And the industry wonders why adoption just won't take...

Still, that's far too simple and straight-forward for me, so continuing our TCP Connected Lighting Gateway programming from the last post, lets cover two of the four commands we use.



To control individual bulbs, you use one of these two commands. The first turns a bulb on and off. The second command will set the dimming level.

def set_light_value(self, did, value=1, update=True):
    rc = self.my_gateway.send_device_command(did, value)
    if update:
    return rc

def set_light_level(self, did, level=100, update=True):
    rc = self.my_gateway.send_device_command_level(did,level)
    if update:
    return rc

And then the HTTPS calls to the gateway are:

The two strings, DeviceSendCommand and DeviceSendLevelCommand were shown back here.

The last two commands of the four affect an entire room and they're used in the same manner:



def set_room_value(self, rid, value=1, update=True):
    rc = self.my_gateway.send_room_command(rid,value)
    if update:
    return rc

def set_room_level(self, rid, level=50,update=True):
    rc = self.my_gateway.send_room_command_level(rid,level)
    if update:
    return rc

No surprise here, the HTTPS calls to the gateway are:

The "update" variable is used to potentially postpone a call to reload the room and bulb status values. That call, the RoomGetCarousel call is a bit 'thick' and I wanted to avoid calling it if I'm doing multiple bulb updates,

(#26) - Let there be light! 

Insufficient coffee to fuel any witticisms or snarky remarks, means I'll cut to the chase. The lighting controller in my house is from a company called TCP Connected. I've discussed the challenges I've had with their product back here.  

Bottom line: it works (it controls the lights well), the company has abandoned the product and I've invested enough in the product that I don't want to abandon it.

Standing on the Shoulders -- Again

Several others out there in 'net land must feel the same way, in that they've reverse engineered the protocol used and resurrected the product. Thanks to folks like this person here, we're wresting control of the lighting gateway from the manufacturer and making the thing useful again.  Thank you!

The Code Chase We're Cutting to (towards?)

If you promise not to make fun of my Python, I'll share the relevant snippets.


The RoomGetCarousel command is the command sent to the Lighting Gateway that returns a lot of information in an XML payload. In this large XML stream, you'll find the name of the rooms, the names of the bulbs, and their state.

[ Nota Bene: You may be able to update the Gateway with new room and bulb information, but I've been doing that using the phone application when connected locally. ]

I have a TCPGatewayManager object who's responsibility is to send the https commands to the gateway

The Gateway supports SSDP but mine's assigned a dedicated IP address so I don't bother with that discovery protocol.

Here's the call to the Gateway using HTTPS:

Parsing the XML payload slowly to discover rooms, bulbs and other neat things:

def load_rooms_and_devices(self):
    logging.debug('reloading rooms and devices from gateway')
    xmldata = self.my_gateway.get_response_room_get_carousel_command()
    root = ET.fromstring(xmldata)
    gwrcmd = root.find('gwrcmd')
    gcmd = gwrcmd.find('gcmd')
    gdata = gwrcmd.find('gdata')
    gip = gdata.find('gip')
    returnCode = gip.find('rc').text

    # Iterate through the rooms and devices
    # Bear in mind that you must send the right XML to the
    # RoomGetCarousel command
    # to retreive all of these fields below.
    self.rooms = []
    self.lights = []
    self.light_names = []
    for a_room in gip.findall('room'):
        r_id = a_room.find('rid').text
        r_name = a_room.find('name').text
        r = {r_name: r_id}

        ## print 'Room ID: ', r_id, ' Name:[', r_name, ']'
        for a_device in a_room.findall('device'):
            a_did = a_device.find('did').text
            a_name = a_device.find('name').text

            a_state = a_device.find('state').text
            if a_state == '1':
               a_device_powered_on = True
                a_device_powered_on = False

            if a_device.find('offline') is not None:
                a_online = False
                a_level = "Unknown"
                a_online = True
                a_level = a_device.find('level').text
            ## print '   Name:[', a_name, ']  DID:', a_did, '  
            ## Powered On:', a_device_powered_on, 
            ## '  Connected To GW:', a_online, '  Level:', a_level

            a_bulb = {'name': a_name, 'did': a_did, 'level': a_level, 
                      'on': a_device_powered_on,
                      'connected': a_online, 'rname': r_name, 
                      'rid': r_id}

            self.light_names.append({a_name: a_did})

I'm still playing with the Python structures that hold my return values. Right now I care about: rooms (their names and numeric IDs) and bulbs (their name, their Device ID [did], their state and their room).

Next post will cover how to turn the lights on, off and dim them.

Tuesday, June 20, 2017

(#25) My IoT Garage Door Closer

There's a big problem with leaving your garage door open at night.

And the problem is that the thieves never take the stuff you want them to take. 

Do they grab the box of outdated college text books from 1985? 

Do they grab the mostly-empty can of Oldham Tudor Earth Tone brown paint you bought in 1995 when that color was in vogue? 

How about the bicycle pump that still works, sort of? No. The really-broken-but-I-can-fix-it lawn chair? Nope.

Then there's a big problem with big problems (in my head) and that's that I tend to avoid the cost-effective and highly reliable solutions out there and pursue a DIY approach.

And there's a big problem with a dearth of clip-art about leaving your garage door open at night. So instead of a nice picture of a wide-open garage door, I'm using a picture of what I can only surmise is a puzzled burglar who came completely equipped with a big bag of break-in tools only to find a house, well lit with windows and doors wide open.

Admiral Gial Ackbar's warning seems appropriate.

I'm not naming names...

So, someone in our house tends to leave the garage door open. And that's a problem. We've already got the sensors installed to detect the open door. All we need to do is provide a means to close it.

The 'net has dozens of posts on how to build a garage door closer with a Raspberry Pi and a relay and I stole one approach from here. Thank you Andrew

My Raspberry Pi 2B controller is wired like this:

5V power to the relay comes from an old, USB charger cable.

Python Relay Control

The python code to control the relay is simple

import RPi.GPIO as GPIO
from time import sleep

class Relay(object):

   def __init__(self):
       print 'Creating and initializing relay class'

       # I've wired door 1 to relay 1 to GPIO pin 23
       # door 2 is relay 2 on GPIO pin 24
       # sleep 0.25 seconds between relay trigger low/high to simulate button press

       self.door1_pin_number = 23
       self.door2_pin_number = 24
       self.latch_time = 0.25

       # Set relay pins as output
       GPIO.setup(self.door1_pin_number, GPIO.OUT, initial=GPIO.HIGH)
       GPIO.setup(self.door2_pin_number, GPIO.OUT, initial=GPIO.HIGH)

   def trigger(self, door_id=1):
       if door_id == 1:
           pin_number = self.door1_pin_number
           pin_number = self.door2_pin_number

       print 'Relay Trigger invoked on pin number', pin_number
       GPIO.output(pin_number, GPIO.LOW)
       GPIO.output(pin_number, GPIO.HIGH)

   def cleanup(self):
       print 'Cleaning up relay'
       GPIO.output(self.door1_pin_number, GPIO.HIGH)
       GPIO.output(self.door2_pin_number, GPIO.HIGH)

Passing MQTT Messages (AGAIN!)

My Home Automation approached is based on systems passing JSON messages over MQTT. That means it's time to define the JSON messages that'll control our controller.

The Command Packet:

Valid commands are “open”, “close” and “trigger”. The first two do what you'd expect, except if an open door receives and open command – it's ignored. Likewise, a close command to a closed door is ignored. I added the “trigger” command to just activate the relay regardless of the state.

Every five seconds or so, the Status Packet is published:

The first datetime is the current date / time of packet creation. Valid door states are 'Unknown', 'Opened', 'Closed' and 'In-Motion'.

The system object simply sends back some information about the RPi that I'll find useful.

If you look at older blog posts you may notice that there are some changes afoot in my design. Over the years now of accumulated MQTT usage, I've concluded that I want the MQTT topic to be part of the payload. I'll spare you the arguments for/against I've had with myself. But, I want the topic to be part of the payload.

You can see in my older MQTT messages, the topic is outside the JSON payload. I've changed my mind on that too now – the entire MQTT packet payload is a JSON formatted messages and the topic is part of the message.