Finding Events: Top-Down vs. Bottoms Up

Whenever I talk to serious computer professional, there is always the debate of which is better: Top-Down or Bottoms-Up. Of course, like a good consultant, the answer is obvious… it depends.

I think in the general computing realm there are practical reasons for doing it one way or the other, but there is also something to how the minds of different people work. My mind typically likes to think of things from the top down. I ask the question “what do I need to get (or become) what I (or the system) wants to be (or know).” My colleagues who prefer a bottom-ups approach ask “what can I do with what I have”. As I contemplate events, these questions seem germane and both viewpoints separately have value, but combined allows for greater success. So this isn't really a competition between the two viewpoints but rather a recipe for putting things in place to allow the viewpoints to work in tandem.

I’ve been giving a lot of thought to how do we do event processing. There are a number of great books on the topic. (Just a couple of examples: David Luckham (and here), Opher Etzion & Peter Niblett and Roy Schulte and Mani Chandy) There is a lot in these books about how to engineer the event reaction. How the machinery works. This is important because as an industry we want to build systems that react to stimuli and manipulate those event objects. I wrote a blog entry last year called the 4Ds: Detect, Derive, Decide and Do which breaks down the areas of concern for manipulating and reacting to events. However, what I haven’t read much of yet is what is an event metaphorically speaking. Sure, we have the EPTS glossary which defines it. We just sort of know intrinsically. Yet, we still question how do we find them in our organization. As an event practitioner, what should I call an Event in my business and how do I prepare myself to exploit the knowledge of their occurrence?

I won’t be so vain to say what I’m about to give is a methodology; nothing that formal. But these are my tricks of the trade; my mode of thinking. Take it for what it is worth, but I think if you can master this, then the practical stuff (getting the engineering to work) will be much easier and more beneficial.

To understand my mind of thought, I view these below as axioms:

• All events have at least one subject which is the noun(s) whose state has changed.
• What we manipulate within computer systems is not the event itself but rather observations of state changes for nouns that are significant enough to warrant attention.
  
• There maybe multiple observations and multiple observers of the same event and each observation maybe made from a different perspective.
• All state changes to nouns occur as the result of some process; although the process may not be known and/or not in our control.
  

Here it goes:

Top-Down Event Thinking
The top-down mode of thinks starts by saying “I need to know that something happened.” That something could be an order was fulfilled, fraud has occurred, the bridge has reached a critical stress point, etc. One thinks of this event typically because one wants a reaction. As shown in the 4Ds, one wants to do something as a result of the event. When the order was fulfilled, I want to issue a commission check or when the bridge has reached a critical stress point, I want to get maintenance people over there.

The question for the event practitioner is “how do I know that the event occurred?”

There are three ways:

• I can sense it (some instrumentation can create a signal when the event is detected)
• I can be told it (some human or other system says that an event occurred)
• I can derive it (some pattern of events occurred within a sphere of observation which allows that event occurrence to be derived)

The EPTS vocabulary says that the first two are “raw” events and the third is a “derived” event. From a top-down thinking prospective, raw events are kind-of boring. I have some sensor or some outside observer who tells me “something happened” and I react. (To imagine raw events think of a thermometer sensing the room’s temperature or a data entry clerk hitting submit button when the order leaves the building (is shipped))

A derived event, from a top-down thinking perspective, is much more fun. When you can’t directly instrument to detect that something has occurring or be told by some other observer, then you have figure out “what would tell me the same thing?” In my 4D post, I mentioned a use-case involving a toll-road wanting to know if someone is speeding. Police officers with speed guns have a direct instrumentation of a car’s speed except the problem is they can’t be everywhere. In addition, people aren’t going to tell you “I sped”. Instead, the event practitioner will have to figure out a way with the set of available, observable events. In the example case, such an observable event is the time an individual car went through two known locations. So, by determining the elapsed time between these two events observations and knowing the distance between the locations, an average speed can be calculated and compared to the legal limits.

The quest for top-down events usually comes because someone wants some sort of reaction to occur. In every industry there are many times people will say “if only I knew ___, I could ____”. As event practitioners, these statements should be captured and contemplated. Business analysts, architects should be able to accessible that list and the event practitioner should be always trying be probing to understand what raw and derived events are being generated by various projects. One of the event practitioner's key responsibilities is to be an activist and take an enterprise-wide view on events and get the projects/applications to publish events that maybe useful to the enterprise but not yet useful to the individual project.

I suggest having a wiki page (or however you like to organize things) where you can quickly capture the desired events and the corresponding reaction. Organize an index by both the event and the reaction. As your business analysts and architects go through their work, they should be checking against this list. Also, as other events cause the same reaction or other reactions to the same event are discovered, they can be documented.

Multiple levels of non-raw events
It is very important to realize that when you consider a target event (one that you want to react to) that you might be able to determine a pattern of events that will derive your target event, but lack the ability to observe all of the events that make up the pattern. To illustrate this we may know that event A occurs whenever event B and event C occur in order within a certain period of time. However, we don't have a means yet for observing event B. This happens all the time because of an old technique called “divide and conquer.” You’re dividing the problem up into smaller problems and then solving the set of smaller problems.

As the event correlation occurs in more and more layers, you can imagine a tree of events: Event A as the root and its branches show that B and C occur. B and C are also shown by their branches, etc. At the leaves of this tree are raw, directly observable events. So, one of the main jobs of an event practitioner is provide a mechanism to capture these trees and therefore, these event patterns should also have a section in your event catalog.

Bottom’s Up Event Thinking
Bottoms up deals with manipulating the event that I can observe and determining how to use them to give you more information. i.e. I see that every time B and C happen, A occurs.

There are thousands if not millions of events in any organization that occur everyday. Only some of them are collected in a manner than can be manipulated in an event processing system; most are not. What you want to be able to do is to discover which state changes have high value, and archive what they mean, where the occur and in relationship to which nouns and processes. To accomplish this you need to have three things. 1) A catalog of these events that is easy to reference, 2) a very good semantic meaning of the event, 3) a sense of the object (noun) that the event has as it subject.

Catalog of Events
A catalog of events is not something, to my knowledge, that you can purchase directly. To do it correctly it needs to be associated with a good Master Data Management strategy and tooling. But even having this as a wiki or a spreadsheet is leaps better than not. So what do you keep in your catalog. Carefully thinking about this will be the difference between success and delusions of adequacy.

Name: The bard’s question of “what is in a name?” still rings true. The name has to describe in a couple of words what this is all about. I like to use the format Noun_ToStateChange or Noun_RelationshipToStateChange_Object. A couple of examples: CustomerOrder_Received, Wife_Married_Husband. The point is not to make it drawn out but to give a sense for why this is important.

Description: Details, details, details. In this section you should describe from the view point of the consumer of the event what this means. An example for CustomerOrder_Received might be “The order was received by the business. No processing has occurred to fulfill the customer’s order other than its entry in the order entry system. This event will be published regardless of channel receipt of the order was facilitated through.” The idea is to remove any ambiguity that may arise because of the name.

Observer/Publisher: Who/What is responsible for observing and publishing this event. An example for CustomerOrder_Received might be “The system that is the system of record for the order will publish its receipt.”

Notification Channels: This is a list (maybe a list of one) where this event will be published. It should include the type of channel (ESB, WebService, MQ, EPN …) and corresponding information like the topic or queue name. The idea is two fold. To give the places so that potential/required publishers know where to publish as well as potential subscribers.

State Transition Diagram: The State Transition Diagram (STD) should be developed for all of the major nouns of the organization. If it is determined that there should be an event published about this noun, then part of the analysis definitely should be a STD along the noun dimension(s) that the particular state change represent. This field should indicate the location (typically a URL) of the STD.

Good semantic description of the event
The question you are trying to answer is “what does the event mean”. On the surface this seems simple. But different viewpoints are going to think differently about the same occurrence. An example… A customer of mine received a report, we’ll call it report CR, from their clients via several integrators. So they one file may have the report from several clients. My customer published an event “CR report file received”. I was confused when I saw this event. Is it saying the physical file from an integrator was received so therefore the system needs to tear it apart into each client’s CR report or was it publishing one event for every client who had their CR report in that file from the integrator? A good semantic description (meaning) in an accessible place would have cleared my confusion.

The object (noun) the event has as its subject.
One of my base tenets is that all events are state changes of some noun. The noun maybe esoteric, but it is still the noun. Therefore each event has a subject and that subject is noun that the business care enough about to track. As such, there should be in your event catalog an index on the nouns of the organization. A clever company will someday understand the relationship events have with an organization’s nouns which are managed in the master data management system, and therefore this clever company will include an event catalog in their MDM.

Finding Events
Here are some good places to start to capture “events”. High value processes have two important characteristics: they deal with things that are already important to the organization and two because they are processes, we know that something “kicked them off”; find what did. Another good place to find events is the swivel chair integrations that occur. However, the largest stash of event is the ERP system or whatever is the fundamental system that keeps things running.

Events: The story of business

"There is no greater agony than bearing an untold story inside you. " - Dr. Maya Angelou

I read that quote and I think about Event Processing. Your probably thinking? "What are you talking about? What does a story have to do with event processing?"
It is this: The event is the captured story of business.

Event Processing is much more than the making clever observations

There are many people who focus on the mechanics of event processing. They think about mathematically how do we combine/aggregate/pattern recognize these event into something more meaningful. This is an incredibly important for without it, the "magic" is gone. But I feel there is a lack of people teaching about what is an event and how to we find them.

As said in many places, the event is the happening, the instant. Something happens to change the world as we know it. It might be something relatively small like a taking money out of an ATM or something with significant impact like a tsunami. But even small things change, in one context or another, the world as we know it. Event Processing has to do with detecting or deriving those changes in the world as we know it. The magic is not observing the happening, but capturing it in a way that we can react to it in our systems.

But how does one become a magician? Where to start? How do we bridge the gap between the physical domain and the virtual domain?

How do we change our mindset to see events

The main thing I would suggest is not to think about the individual events, but rather think of the story. We humans think in terms of stories. Of linear activities, of plot, of tension and inter-relationships. Is satisfying the customer order not a story? Or buying a stock at the right time? True, it might not even be interesting enough to make a "made for television" movie. But it is still a story. And in these stories are character and things happen to these characters. These characters do things and hand things off to other characters. Sure; you can be unromantic and call it a "business process" but still, it is a story. And in this story, as in all stories, there are natural times of interest where something of note happens. These times of interest are the events...

Think about your favorite movie or book. One of mine would be the Fellowship of the Ring by J.R.R. Tolkien. (perhaps you saw the movie) In this movie, some changes in the world were very obvious. Bilbo Left, Frodo got possession of the Ring, Nazgûl search for Ring, Frodo is stabbed with the Mordor Blade, Elrond heals Frodo to the best of his ability, etc. It is easy to see these are major changes in the state of the world. Of course there are more fine-grain events, but don't worry about them yet. Find the big ones in your story.
The way we find the big events in a movie is imagine someone says "Tell me what happened in the Fellowship of the Ring." You would tell all of, what the screenwriters call, the plot points. The major things. Again, there are different layers to the story. You can always go down a layer, later. But think of the large story plot points in your companies story.
"We get an order, we make a product, we bill the customer, we receive the check, we deposit the check" That, my friends, is a story. There are five events that occurred here. Then you start thinking of other end-to-end stories. "We discover a needed capability, we articulate that in a job description, we determine what we will pay for that capability, we advertise, we interview, we offer the position, we on-board the employee". Each of these plot points or events you should capture on a white board. Start to figure out when that event occurs, what do I know, who is involved, what products are being manipulated, when did it occur. These are the things that will make up the meta-data of your events.
Then ask, how can I capture that this occurred? Is this something that happens in a computer orchestrated process that I simply can publish that it happened? Is there a sensor that can sense the physical domain and determined it happened? Is there a way that I can determine it based on other events? Do I need to have a human put it into some UI because there are no other way, yet, to capture it? Does it come from customer, partner, information feed?
What is neat about these stories is that they are really the processes of our business. These events are the natural segmentation points in the process. So you can ask your question to go to the next layer: "What happens between receiving the check and depositing it?" This gives you more processes and more events, but more importantly, a natural hierarchy of events happens. It will also occur that you realize a story is part of a larger story. Of the ecosystems your company belongs; of the rivalry between you and your competitor. That things are occurring outside of your control that affects your story. These plot points are very important to capture.

How does this relate to Dr. Angelou's quote?
I was in a working group with Roy Schulte when an idea hit me. Events are happening in the bowls of the computer system. They generally happen inside of application code and are only observed by a small subset of applications. So, that program is the only one who can react to it. They are stuck there and are limited to expression by the program, which has been our practice, sending out command and control signals to order systems to do a particular part of the work. But that violates the principle of maximize cohesion and minimize coupling. Command and control couples the issuing system to the receiving system; increasing the complexity and ultimately the usefulness of the ecosystem. If the ecosystem had emotions, would it not be in agony?
With events, we decouple the applications. We give voice to the story outside of one application. We allow others to hear the events of the story and allow them to become another plot-line within the story without requiring the "main application" to become more complex. It allows other systems and capability to arise. To fill in gaps. To be built by others and work in tandem and be purposeful.

How does the story metaphor bring event processing to the masses?

We live and breath stories. Stories have as its foundation a state transition: An equilibrium, the equilibrium is lost because of some change (detect/derive), we trying to figure out how to get back to equilibrium (decide), we act (do), equilibrium is reestablished. When you think about your business processes as the stories of your business, the effort of noticing the large events and drilling down brings on a whole new way of thinking. The masses of business people and business analysts may not get what is happening under the covers to allow the derived events to be observed. But the story is a metaphor that all humans get!

So, tell your story well.

What does everyone keep considering WS-* as SOA?

I was reading Dana Gardner's blog on WOA will soon eclipse SOA as most impactful business transformation agent. I am confused why there is even a challenge.

Even at Gartner's Gartner Application Architecture, Development & Integration Summit Dec 2008 in Las Vegas had two men I highly respect Roy Schulte and Nick Gall (blog) had a point-counterpoint discussion about WS-* vs. REST. Although it was very fun to listen and participate, I believe that when statements about WOA replacing SOA doesn't seem to jive with me.

To me, SOA is more about take functionality and making it abstract, network-able and find-able. I believe that it's goal is to abstract functionality to the level that business people can understand and use it. That is why SOA is an incredible partner with Event Driven Architecture, BPM, BAM, Mash-ups, etc. When a business person (I would dare say the new "programmer") can understand the command of our new paradigm, productivity and effectiveness will go through the roof.

Perhaps the discussion focuses more around which is a better protocol for getting at the information. REST or WS-*. In that context, there is a lot of worthwhile debate. But I don't think an answer either way would make "WOA replace SOA" I think they are on different layers of abstraction.

I read a presentation of Nick Gall's (don't have the reference) and it said "We made software because hardware was too hard. Now software isn't soft enough. What is softer than software?" This is the problem I want to solve. I believe that the coordinated application of EDA (CEP), SOA, BPM, BAM in concert with a business awareness found in Enterprise Architecture will lead us down to the solution of Nick's challenge.