Events: The story of business

"There is no greater agony than bearing an untold story inside you. " - Dr. Maya Angelou

I read that quote and I think about Event Processing. Your probably thinking? "What are you talking about? What does a story have to do with event processing?"
It is this: The event is the captured story of business.

Event Processing is much more than the making clever observations

There are many people who focus on the mechanics of event processing. They think about mathematically how do we combine/aggregate/pattern recognize these event into something more meaningful. This is an incredibly important for without it, the "magic" is gone. But I feel there is a lack of people teaching about what is an event and how to we find them.

As said in many places, the event is the happening, the instant. Something happens to change the world as we know it. It might be something relatively small like a taking money out of an ATM or something with significant impact like a tsunami. But even small things change, in one context or another, the world as we know it. Event Processing has to do with detecting or deriving those changes in the world as we know it. The magic is not observing the happening, but capturing it in a way that we can react to it in our systems.

But how does one become a magician? Where to start? How do we bridge the gap between the physical domain and the virtual domain?

How do we change our mindset to see events

The main thing I would suggest is not to think about the individual events, but rather think of the story. We humans think in terms of stories. Of linear activities, of plot, of tension and inter-relationships. Is satisfying the customer order not a story? Or buying a stock at the right time? True, it might not even be interesting enough to make a "made for television" movie. But it is still a story. And in these stories are character and things happen to these characters. These characters do things and hand things off to other characters. Sure; you can be unromantic and call it a "business process" but still, it is a story. And in this story, as in all stories, there are natural times of interest where something of note happens. These times of interest are the events...

Think about your favorite movie or book. One of mine would be the Fellowship of the Ring by J.R.R. Tolkien. (perhaps you saw the movie) In this movie, some changes in the world were very obvious. Bilbo Left, Frodo got possession of the Ring, Nazgûl search for Ring, Frodo is stabbed with the Mordor Blade, Elrond heals Frodo to the best of his ability, etc. It is easy to see these are major changes in the state of the world. Of course there are more fine-grain events, but don't worry about them yet. Find the big ones in your story.
The way we find the big events in a movie is imagine someone says "Tell me what happened in the Fellowship of the Ring." You would tell all of, what the screenwriters call, the plot points. The major things. Again, there are different layers to the story. You can always go down a layer, later. But think of the large story plot points in your companies story.
"We get an order, we make a product, we bill the customer, we receive the check, we deposit the check" That, my friends, is a story. There are five events that occurred here. Then you start thinking of other end-to-end stories. "We discover a needed capability, we articulate that in a job description, we determine what we will pay for that capability, we advertise, we interview, we offer the position, we on-board the employee". Each of these plot points or events you should capture on a white board. Start to figure out when that event occurs, what do I know, who is involved, what products are being manipulated, when did it occur. These are the things that will make up the meta-data of your events.
Then ask, how can I capture that this occurred? Is this something that happens in a computer orchestrated process that I simply can publish that it happened? Is there a sensor that can sense the physical domain and determined it happened? Is there a way that I can determine it based on other events? Do I need to have a human put it into some UI because there are no other way, yet, to capture it? Does it come from customer, partner, information feed?
What is neat about these stories is that they are really the processes of our business. These events are the natural segmentation points in the process. So you can ask your question to go to the next layer: "What happens between receiving the check and depositing it?" This gives you more processes and more events, but more importantly, a natural hierarchy of events happens. It will also occur that you realize a story is part of a larger story. Of the ecosystems your company belongs; of the rivalry between you and your competitor. That things are occurring outside of your control that affects your story. These plot points are very important to capture.

How does this relate to Dr. Angelou's quote?
I was in a working group with Roy Schulte when an idea hit me. Events are happening in the bowls of the computer system. They generally happen inside of application code and are only observed by a small subset of applications. So, that program is the only one who can react to it. They are stuck there and are limited to expression by the program, which has been our practice, sending out command and control signals to order systems to do a particular part of the work. But that violates the principle of maximize cohesion and minimize coupling. Command and control couples the issuing system to the receiving system; increasing the complexity and ultimately the usefulness of the ecosystem. If the ecosystem had emotions, would it not be in agony?
With events, we decouple the applications. We give voice to the story outside of one application. We allow others to hear the events of the story and allow them to become another plot-line within the story without requiring the "main application" to become more complex. It allows other systems and capability to arise. To fill in gaps. To be built by others and work in tandem and be purposeful.

How does the story metaphor bring event processing to the masses?

We live and breath stories. Stories have as its foundation a state transition: An equilibrium, the equilibrium is lost because of some change (detect/derive), we trying to figure out how to get back to equilibrium (decide), we act (do), equilibrium is reestablished. When you think about your business processes as the stories of your business, the effort of noticing the large events and drilling down brings on a whole new way of thinking. The masses of business people and business analysts may not get what is happening under the covers to allow the derived events to be observed. But the story is a metaphor that all humans get!

So, tell your story well.

The 4Ds "Detect, Derive, Decide and Do"

The Interesting thing about the 4Ds

As I am sure all of us are familiar, it is in the framing of the problem that gives the greatest ability to abstract the problem.

Thus the framing where the 4Ds come into play.

Detect
This is the act of capturing the stimuli that comes from an outside source. This might be a sensor reading or a request to do some action.

The essence is moving it from one domain to another domain. In the case of temperature control, the heat sensor allows the capture of a stimuli of the "reality domain" to the "device domain". It may also be more virtual. I would argue that a web service call from one partner to another partner is the capture of the "request stimuli".

Included in this "area" is the conversion of the stimuli from the raw datum to a virtual version of that stimuli.

Note: You will see that I refer a lot to the stimuli. I do this because I believe that a service provider is constantly capturing stimuli in which to react. Perhaps even reacting to the lack of stimuli.

Derive
This is the act of correlating this stimuli with other stimuli or other derived understanding of the world. So what's does that mean?

When you think about situational awareness, it is basically an understanding of the world within the scope of context you care about. A big state machine in which reality (as it can be captured) is made virtual. Hmm.

CEP plays a large part in this because you can take various stimuli (called events and state changes), combine/aggregate/pattern recognize/intersect/unionize/... them and derive other stimuli (also called derived events). The power of which is events (happenings in a different domain) can be detected that can not be directly sensed in that other domain. Said differently, these events (or stimuli) that occur in a different domain can not be directly detected because there isn't a mechanism to capture the stimuli. However, by looking at patterns of events that you can detect in concert with other captured events, you are able to derive events in other domains that are not directly detectable . Wow! That's powerful stuff!

Let me give an example to illustrate this.

Let suppose that there exists (and there does) a camera that is able to take the picture of a car's license plate as it goes through a toll booth. This, obviously, would represent detect. This sensor takes a picture, converts the images of the plate and converts it to a variable that contains the letters and numbers of the license plate. As well, it includes the meta-data of that datum. In this case, the time and location that the picture was taken.

After some time, the car passed through another toll booth and another detect activity occurs. Except in this case, the meta-data of the license plate number datum has a different time and location. (I am assuming that the guy didn't just spin around the toll booth!)

Both of these events (stimuli) have been detect activities. Now comes the fun part.

Because the same entity owns both toll booths, it is able to compare the two detect events and derive a new piece of information. The elapsed time the motorist took between the two toll booths. This organization could then divided the distance (which they would know) by the elapsed time would would yield the average speed of the car associated with the license number between the two toll booths. This information would be added to meta-data about the trip which is a higher order abstraction that the passing of a toll booth. ( I am assuming that this isn't James Bond who has license plates that can swivel to a new one )

So, the adding of information or determining situational awareness is the role of the derive. I want to derive more information about the trip the license plate took.

More specifically, the determination that the motorist was speeding (specifically violating the law that regulates speed of motorists along that road) would also be derived.

Decide
The decide section, as its name suggests, is determining if and what to do because of the awareness of the situation.

In our example, the rules (decision logic) may specify that if the speed was over a certain tolerance, then we issue a ticket, if it is less than that tolerance, we ignore the stimuli.

Business rules engines are very good at handling this capability in a very configurable manner . But just as likely a case statement or IF THEN ELSE might represent the decide action.

Do
Again, being the master of the obvious, the name of this section represent doing the activity that was decided to be done.

In the case of the traffic example, it might issue a request to the ticketing web service to cite the offending motorist or that capability might be within the boundaries of the service and would execute its own mechanism.

How the 4Ds solves the Problem
Now that the 4Ds are understood and the offending motorist has contributed to the city's budget, your probably wondering, so how does this help me develop a solution?

With all abstractions, we are looking for ways to divide the problem. But not just for divide and conquer purposes. There are different mindsets and concerns that each of these areas would require. And potentially different expertise. A person that knows how to manipulate the sensor to take the readings dealing with harmonics and weight levels for a bridge may not necessary know the statistical equations to put those data points together to determine the bridge's stress or decay. Similarly that individual or team may not decide how we react to those stress levels. Nor are they necessarily the ones who will put the mechanics of put into the action the decisions. Perhaps they are, but this solution allows these factors to be separated.

Does this go beyond Event Processing and SOA
As I said earlier, I don't believe this is limited to simply CEP or expert systems. This is the nature of a reactive system, which all computer systems in truth are. Even "proactive" systems are reacting to an understand of the world around us and a potential threat or opportunity (also known as situational awareness). The difference, in my mind, between a proactive and a reactive system is a proactive system is one where the actions occur before that HAVE to be or for marketing purpose before the competition reacts to it.


Summary
So, in summary, I hope that I have given you reason to 1) think about systems as reactions to stimuli (events (changes in the world around us)) and 2) that when we break the solution up into the 4Ds, we are dividing the solution into logical segments that takes advantage of the discrete types of thinking.

Notes:
As always, I want to have a humble ego when it comes to my ideas. I realize that no man or idea is an island unto itself, but rather stands upon the shoulders of giants. I hope that if anything, it acts as a springboard for others to have even better ideas or acts to help enhance or dismiss my ideas into something more accurate and truthful. So, I will take any comments as constructive criticism.