(#18) More Esper Tidbits
As I continue to use and explore the capabilities of Esper, I'm going to put down some of the answers to questions that I ran into.
The documentation in Esper is certainly comprehensive but, to me, it's not as approachable for a novice, as I think it could be.
So I'm coding, playing and using the Netbeans debugger to see how things work.
Note, take all of the code snippets with a grain of salt. I'm not claiming them to be correct nor proper nor efficient. They're just examples of my discoveries.
Recall that I'm using Esper to be the CEP engine for a Home Automation project. My sensors around the house emit MQTT packets periodically. There are door sensors, motion sensors, weather sensors, a Nest Thermostat, a Caller ID box and so on.
Esper's job is to take the MQTT packet data from the sensors and try to draw conclusions about what's going on in the house. Did someone just leave? Did someone just come home? Did my spouse just crank up the thermostat to some hellish level and single-handily impact, adversely, the Natural Gas stores in Colorado?
While the Esper documentation is great about providing examples on the EPL, I've also been struggling with what the handler code should do? The handler, "listener" in Esper parlance, is the code that's invoked when the EPL finds a match.
Esper Listener Examples
Q: For the EPL "select count(*)..." what does the listener code look like?
A: The attribute to get is named "count(*) and is of type long
For example:
String subQuery1 = "SELECT COUNT(*) FROM WS2308WeatherStatusEvent wse WHERE wse.oTemp < cast(50.0, double)";
The Listener:
@Override
public void update(EventBean[] newData, EventBean[] oldData)
{
Long number = (Long) newData[ 0 ].get( "count(*)" );
An easier way would probably have been to modify the select clause:
String subQuery1 = "SELECT COUNT(*) as numEvents FROM WS2308WeatherStatusEvent wse WHERE wse.oTemp < cast(50.0, double)";
The cast is there because I didn't just default all floating point types in Java to double. I used floats too. Without the cast, Esper complained, at compile time, about the type mismatch.
Q: The Solution Patterns document, under the section "How do I detect N events in X seconds" has the example: "select count(*), window(*) from MyEvent(somefield = 10).win:time(3 min) having count(*) >= 5 output first every 3 minutes".
If I change it to reflect my needs:
anEPLQuery = "SELECT COUNT(*), WINDOW(*) FROM NestStatusEvent( hvacStatus = 'HEATING' ).win:length(5) HAVING COUNT(*) >= 5 OUTPUT FIRST EVERY 5 MINUTES";
then what does the Listener code need to do?
A: Here's what's working for me. To serve as a tutorial, the snippet takes no shortcuts. Some of the statements could be combined.
class FurnaceRunningListener implements UpdateListener
{
@Override
public void update(EventBean[] newData, EventBean[] oldData)
{
logger.info( "FurnaceRunningListener - update called" );
try {
//
// When the EPL is simple "SELECT * FROM EventObject" - you
// use the getUnderlying() method. For example,
// NestStatusEvent eventOne =
// (NestStatusEvent) newData[ 0 ].getUnderlying();
//
// For this, More complex EPL
// "SELECT COUNT(*), WINDOW(*) FROM
// NestStatusEvent( hvacStatus = 'HEATING' ).win:time( 5 min )
// HAVING COUNT(*) >= 5 OUTPUT FIRST EVERY 5 MINUTES";
// Now EventBean[] is coming in with a hashmap
EventType et = newData[ 0 ].getEventType();
String propertyName1 = et.getPropertyNames()[ 0 ]; // "count(*)"
String propertyName2 = et.getPropertyNames()[ 1 ]; // "window(*)
// propertyName1 is "count(*)" - get the value
long countValue = ((Long) (newData[0].get( propertyName1 ))).longValue();
// propertyName2 is "window(*)" - the events are coming in as an array
NestStatusEvent[] triggeringEvents = (NestStatusEvent[]) newData[0].get( propertyName2 );
// How many events? Our EPL asked for (at least) 5
int numEvents = triggeringEvents.length;
// so the first event object is at [0], the last at [4]
NestStatusEvent firstEvent = triggeringEvents[ 0 ];
NestStatusEvent lastEvent = triggeringEvents[ numEvents - 1 ];
//
And go on from there.
BPM - Getting it off my Chest: Microservices and APIs
I mentioned in post 17, that I had come back from a conference on Application Architecture. Hosted by a national vendor with a good reputation for quality. I'm expecting a call from the account rep asking me how I enjoyed the conference. I think what I'm simply going to say is that "the coffee was good."
And leave it at that.
My first criticism was simply that the conference was more "bread than meat" -- the topics were only covered in the most cursory manner. They were all the proverbial 50,000' fly-by.
I get it - the sessions are 45 minutes; you cannot get into too much detail in 45 minutes. But still, that wasn't long enough for me. I don't think I'll go back to this conference because of this alone.
But there was something else at this conference. What's a polite synonym for pandering? By the way, BPM, for this post, does not stand for Business Process Management.
Buzzwords de' decade.
Microservices and APIs. Ugh. Every damned slide, poster, badge, ornament and coffee cup holder was emblazoned with Microservices and APIs.
Really? Really? Really? Give me a break.
So just what is an API anyway?
This one was puzzling. We've been defining, designing, implementing and using APIs for over 40 years. What was so different about the 2014 definition to warrant all this hype?
What's in shared library or DLL? An API. How did the client / server applications we created in the 80s interact? Through APIs. Apollo Network Computing, OSF DCE, CORBA, SOA -- all laden with APIs.
So what's the difference between a service and an API anyway? Only one analyst tackled this one head on and answered "Nothing." Thank you!
I'd venture a guess that that vast majority of attendees didn't know or didn't care about this. I did. Is it just the malcontent of an old curmudgeon? I don't think so. I walked out feeling like I was being hyped.
I expect this kind of hype from a vendor. But I thought it quite beneath the analysts at this conference.
I found it distasteful.
Microservices
Goodness I don't know where to start. Maybe I'll start off with a couple of opinions.
Opinion Number 1 - There's nothing new here
Service based design and programming is as old as dirt.
Ok - as old as computer dirt. I stumbled across the early trails, then called Network Computing, in the mid 80's with such offerings as Sun's RPC/XDR, and NFS products.
A few years later and the design approach had gathered enough of a following to warrant OSF's DCE offering. So it's at least 35 years old, and probably older.
Again - this isn't a rant about the old farts not getting credit for something. This is a warning. Services, micro or macro, are not new.
The service-based shoals of application development are littered with the bodies of those who've tried and failed to make these things work. Which leads to Opinion Number 2.
Opinion Number 2 - There were 54 problems we ran into when designing applications using services. Microservices takes that number down to 53.
When creating non-trivial applications using services we ran into:
- language independent service discovery
- transaction management
- auditing / logging
- orchestration / choreography
- common context (enterprise object models)
- contractual obligations / promises
- testing
- granularity
- security
- error handling
And 43 others. Just Google "SOA implementation problems" for a taste of what we ran into.
With microservices there's a half-hearted attempt to tackle one: granularity. Make 'em small. How small? Can't tell you exactly. Make 'em small enough but not too small. Make 'em just the right size.
Sweet. Thanks for the help.
You want to make a real impact - tackle error handling. That should be simple enough.
Bull Hockey
One analyst tossed up a slide showing that "applications of the future will be an assembly of microservices." It was difficult to remain seated. I've seen that slide, in one form or another, for 20+ years.
Ok - show me.
Show me one non-trivial application constructed from an assembly of microservices.
Show me one, non-trivial, business relevant, customer facing application:
- assembled from 6 or more microservices
- with microservices created by 2+ vendors
Sigh.
Never mind...
So, what? Give up?
When I was a kid, about 7, I came to the conclusion that we had invented swearing. My generation had invented cursing. I didn't hear it at home (much) but there was this 5th grader, Jeremy, that could make a sailor blush.
Since that was my first real exposure to the art of cussing, I assumed that Jeremy, the 5th grader, had started it all.
Around 2005, give or take, I had a young gun from Microsoft come into work for a presentation (on BizTalk, I think) and state categorically that Microsoft had invented distributed computing.
If this is your first exposure to service based development, you might think you're blazing uncharted territory too. That's fine. Please let me repeat: this isn't the malcontent of an old curmudgeon who feels like we didn't get our share of credit. I personally had nothing to do with the science, the design of service based design. I just hopped on the bandwagon.
If I have seen further it is by standing on ye sholders of Giants.
Do your homework. Don't make the same mistakes or run needlessly into the same obstacles we did.
Don't give up. Keep working on it.
But I'll close with a suggestion that you keep a George Santayana quote in mind about remembering the past.
Good luck.
May you have more success that we did.
(#17) Well That was Unexpected - CEP Legitimized?
I'm at a Nerd Conference in Vegas. There's about 2000 nerds out here. Mostly IT nerds. IT nerds from big companies. IT nerds from big companies talking about problems they have, that I used to have, that I no longer have.
And that makes me smile.
Much to my surprise, Event Processing made the bill. Not as high on the charts as, oh say, Microservices and APIs. But there were three or so sessions dedicated to event processing in business. The TLA CEP (Complex Event Processing) was scattered on a few slides too.
And that reminded me I hadn't come back to close off the situation presented in post #16. In that post, I said that I had not been getting the results I had expected, nor wanted from Esper's CEP Engine. When I posted my question to the Esper discussion groups, there were several responses saying I should abandon my EPL query and switch over to Match Recognize.
The Match Game
I don't know the history of this syntax. You'll find references to "match recognize SQL" in the documentation for Oracle 12c DBMS. Whether Esper borrowed it or if it's a extension that Oracle came up with or if it's an emerging standard -- I don't know. What I do know is that it worked.
Here's the new query using the Esper's Match Recognize syntax:
anEPLQuery = "SELECT * FROM HHBAlarmEvent " +
" MATCH_RECOGNIZE (" +
" MEASURES A as a, B as b, C as c" +
" PATTERN (A B C) " +
" DEFINE " +
" A as A.macAddress = '000000B357' and A.deviceStatus = 'OPEN', " +
" B as B.macAddress = '0000012467' and B.deviceStatus = 'OPEN', " +
" C as C.macAddress = '000007AAAF' and C.deviceStatus = 'MOTION'" +
")";
EPStatement arrivalStatement_EPL4 = cepAdm.createEPL( anEPLQuery );
Stare at it a bit, and you can kind of tease it out. The Pattern we're going to recognize is (A B C). The pattern syntax is regex based. I'm looking for Event A, then Event B, then Event C. The Measures A as a, B as b means that my listener method can pull out the events using "a", "b" and "c". Like this:
HHBAlarmEvent eventOne = (HHBAlarmEvent) newData[ 0 ].get( "a" );
HHBAlarmEvent eventTwo = (HHBAlarmEvent) newData[ 0 ].get( "b" );
HHBAlarmEvent eventThree = (HHBAlarmEvent) newData[ 0 ].get( "c" );
And the DEFINE part tells the Engine how to recognize a match. It defines the condition that triggers a match. In my case, Event A is when the Garage Door Sensor (MAC Address '0B357') goes to device status OPEN. Then Event B is when the Door to the Garage (MAC Address '12467' also goes to status OPEN. And finally, when those two are followed by Event C, the Motion Detector (MAC Address '7AAAF) detects MOTION.
I'll post the results of this change from Esper EPL (A -> B -> C) to Match Recognize later, but it worked!
I started getting the results I thought I should get.
The syntax for Match Recognize looks a bit harder to master, but it's going to be worth mastering.
