A new column to the Zachman Framework.. "Because"

VA's Version of the Zachman Framework

I think about John Zachman and the Zachman framework and I marvel at how it transformed IT and gave a framework to start discussing what are the elements that are in our computer architecture and it blends in the business viewpoint.  However, Zachman, and most other Enterprise Architecture frameworks began their life as ways of understanding what we build in information technologies.  I am changing my efforts to considering how does a move to an ecosystem computing affect this.

One of the elements of this ecosystem is events and its corresponding event processing.  Events represent "something of significance", so I was curious and wondered "where do events fit on Zachman's Framework"?

The most obvious answer is that there is a column called "when" and some representation of the framework emphasis events.  However, my read of the "when" column is that they are speaking more of knowing the cycles that happen inside an enterprise.  Examples of these cycle are things like "payroll cycle", "sales cycle", "tax season".  They represent the patterns of peaks and valleys of work to be done by the enterprise.  To give another example, a college admissions office may have large spike in work 4 to 5 months prior to graduation for secondary education but have smaller spikes around the start of other semesters.  These cycle are known "way we do business".  Anyone who has a friend who is a tax account in the U.S. knows not to invite them to a party between March and April 15.

What I want to focus on is the events or "somethings of significance" that their occurrence is not known.  These would be things like "competitor released new product" or "stock market gain / loss" or "employee quit"  These things are not schedule events like what you find in the "when" column but they are very critical to the success of the business.  So, why isn't there a means of capturing them?  The column name is "because" as in "I am executing this process because of this situation".

So what would the rows look like for this column?

Perspetive	Because Column	What it means
Executive	List of Situations	What situations do we care about.
Business Mgmt	Situation Definition	What does the situation look like?
Architect	Situation Representation	How do we know the situation occurred?
Engineer	Event Definition	Describe the message used to announce
Technician	Event Algebra	Combination of items used to sense the situation

Types of Processes

Although processes are covered in the Zachman Framework, I wanted to bring something to the attention of  event practitioners.  When I think of processes, I tend to think of 5 types...

1. Processes that we (the company) control
2. Processes that our partners (supply chain) control
3. Processes that our competitors control
4. Processes that government (regulations) control
5. Processes that nature (don't mess with mother nature) controls

As we go down the list, the amount of control is less.  We need to deploy our sensors and attune them to "see" the effects of these processes to gain situational awareness and therefore be able to appropriately react.  It is this space where an event practitioner should focus to bring the most value to the organization.  

Events:The Great De-coupler - Determine the Independance score of an event system.

Events are the great de-coupler because their very nature changes our prospective from command-oriented computing to situation-oriented.

Think about it. Most of the things we have learned in CIS courses was how to help the computer issue commands. It is easier, in some senses, to just command the reaction that we want rather than put it in context so that the situation can be understood by people who didn't write the event publisher. When we simply command the reaction we want, we are increasing coupling. We are tying a dependency from the application that raised the event to the application that reacts to it. I contend that by properly creating the event object with the appropriate references and ontology that support those references that we can decouple our events and our applications even further! We can even go to the point of reacting to events that are created outside of our sphere of influence automatically.

A goal of the event processing community should be to make semantically-aware events that can be consumed by an application or engine with no knowledge of or organizational affiliation to the event's publisher.

To start off, we need to design events to live outside of a particular application, department and organization and even an particular industry. We have already taken strides when various taxonomies and ontology for industry are created. They take the real-world nouns, processes, and state-changes of the industry and give it a namespace so that all may call things the same. This trend promises to continue and become better at defining them.

The table below defines the various levels of independence for an event object. This isn't just about how do we create a good event object that the system can use. This is about building an ecosystem of information that surrounds the set of events for a given organization giving it a rich environment to influence.

For reference sake:

• Event: The change of state of a noun.
• Event Object: A container that encapsulates the meta-data associated with an observation of an event

Category	Description	Comment
Decoupled	The event object [EO] must exist independent of any computing algorithm that would operate upon it's content. Must be independent of reactions to it.	This leaves algorithm that manipulates the container, moving it, etc. This doesn't mean that things can't operate on its content, just that the event object doesn't rely on the operation for its existence. (Obvious exception is the event creation algorithm)
Subscribe-able	A mechanism must exist so that a subscription of the event object can be requested and the event object delivered upon publication.	This is the classic event-driven architecture.
Descriptive not prescriptive	The event object must indicate what happened (state change/action/lack of state change) and not indicate how to react to what happened. An event object must be independent of commands	This one separates out awareness from commands meaning that an event object should describe the situation not issue a command. There are a lot of messages that are decoupled and subscribe-able but the message issues a command or issues constraints to the reaction to a notification. These would not satisfy this level of events.
Contextual	EO must indicate/identify the noun (object) the event is in reference to EO must indicate the state that the noun changed to EO may indicate the state of the noun that the noun changed from An event object must be independent of embedded knowledge in the ecosystem.	This basically says that an event must self-containing it's direct references. The Event Object should answer the questions: What happened? To whom? When? Where? What are the details? What is the certainty?... All of these are referenced inside the event object.
Referenced	EO references must be governed by a taxonomy/ontology An event object must be independent of semantic ambiguity	Referenced means that all pieces of context are referenced to some source outside the event. Similar to Fielding's REST concepts. It should be a URI or something that can identify the subject.
Dimensional	Nouns have defined dimensions that are made up of defined states or values. Since events are an observation of the change of state for a noun, the referenced taxonomy should have a well defined dimensions, states, and values for all nouns under its jurisdiction.	In my view, nouns have dimensions (attributes) that either describe a state or a property. In this maturity, the dimensions must be defined as well as what legal state/value can exist in that dimension.
Cross-referenceable	The taxonomies must be systemically translatable to other taxonomies causing semantic interdependence.	By developing cross-referenced taxonomies, organization outside of the sphere of influence of the event publisher can understand and utilize the events without requiring human intervention.

Finding Events: Top-Down vs. Bottoms Up

Whenever I talk to serious computer professional, there is always the debate of which is better: Top-Down or Bottoms-Up. Of course, like a good consultant, the answer is obvious… it depends.

I think in the general computing realm there are practical reasons for doing it one way or the other, but there is also something to how the minds of different people work. My mind typically likes to think of things from the top down. I ask the question “what do I need to get (or become) what I (or the system) wants to be (or know).” My colleagues who prefer a bottom-ups approach ask “what can I do with what I have”. As I contemplate events, these questions seem germane and both viewpoints separately have value, but combined allows for greater success. So this isn't really a competition between the two viewpoints but rather a recipe for putting things in place to allow the viewpoints to work in tandem.

I’ve been giving a lot of thought to how do we do event processing. There are a number of great books on the topic. (Just a couple of examples: David Luckham (and here), Opher Etzion & Peter Niblett and Roy Schulte and Mani Chandy) There is a lot in these books about how to engineer the event reaction. How the machinery works. This is important because as an industry we want to build systems that react to stimuli and manipulate those event objects. I wrote a blog entry last year called the 4Ds: Detect, Derive, Decide and Do which breaks down the areas of concern for manipulating and reacting to events. However, what I haven’t read much of yet is what is an event metaphorically speaking. Sure, we have the EPTS glossary which defines it. We just sort of know intrinsically. Yet, we still question how do we find them in our organization. As an event practitioner, what should I call an Event in my business and how do I prepare myself to exploit the knowledge of their occurrence?

I won’t be so vain to say what I’m about to give is a methodology; nothing that formal. But these are my tricks of the trade; my mode of thinking. Take it for what it is worth, but I think if you can master this, then the practical stuff (getting the engineering to work) will be much easier and more beneficial.

To understand my mind of thought, I view these below as axioms:

• All events have at least one subject which is the noun(s) whose state has changed.
• What we manipulate within computer systems is not the event itself but rather observations of state changes for nouns that are significant enough to warrant attention.
  
• There maybe multiple observations and multiple observers of the same event and each observation maybe made from a different perspective.
• All state changes to nouns occur as the result of some process; although the process may not be known and/or not in our control.
  

Here it goes:

Top-Down Event Thinking
The top-down mode of thinks starts by saying “I need to know that something happened.” That something could be an order was fulfilled, fraud has occurred, the bridge has reached a critical stress point, etc. One thinks of this event typically because one wants a reaction. As shown in the 4Ds, one wants to do something as a result of the event. When the order was fulfilled, I want to issue a commission check or when the bridge has reached a critical stress point, I want to get maintenance people over there.

The question for the event practitioner is “how do I know that the event occurred?”

There are three ways:

• I can sense it (some instrumentation can create a signal when the event is detected)
• I can be told it (some human or other system says that an event occurred)
• I can derive it (some pattern of events occurred within a sphere of observation which allows that event occurrence to be derived)

The EPTS vocabulary says that the first two are “raw” events and the third is a “derived” event. From a top-down thinking prospective, raw events are kind-of boring. I have some sensor or some outside observer who tells me “something happened” and I react. (To imagine raw events think of a thermometer sensing the room’s temperature or a data entry clerk hitting submit button when the order leaves the building (is shipped))

A derived event, from a top-down thinking perspective, is much more fun. When you can’t directly instrument to detect that something has occurring or be told by some other observer, then you have figure out “what would tell me the same thing?” In my 4D post, I mentioned a use-case involving a toll-road wanting to know if someone is speeding. Police officers with speed guns have a direct instrumentation of a car’s speed except the problem is they can’t be everywhere. In addition, people aren’t going to tell you “I sped”. Instead, the event practitioner will have to figure out a way with the set of available, observable events. In the example case, such an observable event is the time an individual car went through two known locations. So, by determining the elapsed time between these two events observations and knowing the distance between the locations, an average speed can be calculated and compared to the legal limits.

The quest for top-down events usually comes because someone wants some sort of reaction to occur. In every industry there are many times people will say “if only I knew ___, I could ____”. As event practitioners, these statements should be captured and contemplated. Business analysts, architects should be able to accessible that list and the event practitioner should be always trying be probing to understand what raw and derived events are being generated by various projects. One of the event practitioner's key responsibilities is to be an activist and take an enterprise-wide view on events and get the projects/applications to publish events that maybe useful to the enterprise but not yet useful to the individual project.

I suggest having a wiki page (or however you like to organize things) where you can quickly capture the desired events and the corresponding reaction. Organize an index by both the event and the reaction. As your business analysts and architects go through their work, they should be checking against this list. Also, as other events cause the same reaction or other reactions to the same event are discovered, they can be documented.

Multiple levels of non-raw events
It is very important to realize that when you consider a target event (one that you want to react to) that you might be able to determine a pattern of events that will derive your target event, but lack the ability to observe all of the events that make up the pattern. To illustrate this we may know that event A occurs whenever event B and event C occur in order within a certain period of time. However, we don't have a means yet for observing event B. This happens all the time because of an old technique called “divide and conquer.” You’re dividing the problem up into smaller problems and then solving the set of smaller problems.

As the event correlation occurs in more and more layers, you can imagine a tree of events: Event A as the root and its branches show that B and C occur. B and C are also shown by their branches, etc. At the leaves of this tree are raw, directly observable events. So, one of the main jobs of an event practitioner is provide a mechanism to capture these trees and therefore, these event patterns should also have a section in your event catalog.

Bottom’s Up Event Thinking
Bottoms up deals with manipulating the event that I can observe and determining how to use them to give you more information. i.e. I see that every time B and C happen, A occurs.

There are thousands if not millions of events in any organization that occur everyday. Only some of them are collected in a manner than can be manipulated in an event processing system; most are not. What you want to be able to do is to discover which state changes have high value, and archive what they mean, where the occur and in relationship to which nouns and processes. To accomplish this you need to have three things. 1) A catalog of these events that is easy to reference, 2) a very good semantic meaning of the event, 3) a sense of the object (noun) that the event has as it subject.

Catalog of Events
A catalog of events is not something, to my knowledge, that you can purchase directly. To do it correctly it needs to be associated with a good Master Data Management strategy and tooling. But even having this as a wiki or a spreadsheet is leaps better than not. So what do you keep in your catalog. Carefully thinking about this will be the difference between success and delusions of adequacy.

Name: The bard’s question of “what is in a name?” still rings true. The name has to describe in a couple of words what this is all about. I like to use the format Noun_ToStateChange or Noun_RelationshipToStateChange_Object. A couple of examples: CustomerOrder_Received, Wife_Married_Husband. The point is not to make it drawn out but to give a sense for why this is important.

Description: Details, details, details. In this section you should describe from the view point of the consumer of the event what this means. An example for CustomerOrder_Received might be “The order was received by the business. No processing has occurred to fulfill the customer’s order other than its entry in the order entry system. This event will be published regardless of channel receipt of the order was facilitated through.” The idea is to remove any ambiguity that may arise because of the name.

Observer/Publisher: Who/What is responsible for observing and publishing this event. An example for CustomerOrder_Received might be “The system that is the system of record for the order will publish its receipt.”

Notification Channels: This is a list (maybe a list of one) where this event will be published. It should include the type of channel (ESB, WebService, MQ, EPN …) and corresponding information like the topic or queue name. The idea is two fold. To give the places so that potential/required publishers know where to publish as well as potential subscribers.

State Transition Diagram: The State Transition Diagram (STD) should be developed for all of the major nouns of the organization. If it is determined that there should be an event published about this noun, then part of the analysis definitely should be a STD along the noun dimension(s) that the particular state change represent. This field should indicate the location (typically a URL) of the STD.

Good semantic description of the event
The question you are trying to answer is “what does the event mean”. On the surface this seems simple. But different viewpoints are going to think differently about the same occurrence. An example… A customer of mine received a report, we’ll call it report CR, from their clients via several integrators. So they one file may have the report from several clients. My customer published an event “CR report file received”. I was confused when I saw this event. Is it saying the physical file from an integrator was received so therefore the system needs to tear it apart into each client’s CR report or was it publishing one event for every client who had their CR report in that file from the integrator? A good semantic description (meaning) in an accessible place would have cleared my confusion.

The object (noun) the event has as its subject.
One of my base tenets is that all events are state changes of some noun. The noun maybe esoteric, but it is still the noun. Therefore each event has a subject and that subject is noun that the business care enough about to track. As such, there should be in your event catalog an index on the nouns of the organization. A clever company will someday understand the relationship events have with an organization’s nouns which are managed in the master data management system, and therefore this clever company will include an event catalog in their MDM.

Finding Events
Here are some good places to start to capture “events”. High value processes have two important characteristics: they deal with things that are already important to the organization and two because they are processes, we know that something “kicked them off”; find what did. Another good place to find events is the swivel chair integrations that occur. However, the largest stash of event is the ERP system or whatever is the fundamental system that keeps things running.

Cloud-o-gram

When I first saw Brenda Michelson's post Cloud-o-gram, I couldn't help think about sneaking downstairs after my parents went to sleep to watching Saturday Night Live. One episode they had was this land-shark that would try to get into people's houses or apartments by pretending to be a candy-gram delivery. But unlike the delivery of the most cleverest of all sharks, this cloud-o-gram reference architecture seems to be the real deal.

One of the fundamental problems facing things that allow ecosystems to develop on it is how do you segment it into areas so that you can focus your attention? When you do, what lines do you draw so that your segmentation isn't arbitrary? When I look at the cloud and start to talk about it to people (and sometimes just to myself) I fluctuate between customer POV, Provider POV, Ecosystem POV, Machine POV and land-shark POV. And when you add recursion that a provider of a service is really reselling a service that is consuming yet another service which that same physical service might actually be sold by yet another provider! Oy Vey!

My read is that Brenda has nicely separated out what is sold (offerings) from the mechanisms that provide what is sold (cloud computing environment) from the sales management of what is being sold (For the/About the cloud) from the agreement to sell (SLA). Genius!

This reference architecture will be the organizing mindset behind the Elemental Link's research (and I suspect a book or two). What I am unsure of at this point is does the reference architecture allow for multiple points of view? As with most services, the veil of encapsulation allows the consumer to "ignore the man behind the curtain", but to their own detriment in many cases. The power of abstraction hides complexity (as it should) but also increases fear of the unknown or un-understood. This is why what Brenda is offering is important for consumers and providers alike. To dispel the magic to allow rational decisions to be made.

As much as I am glad that she is taking on this important work, I feel that there is a significant area missing: customer reasoning. As with most services, this model also has a boundary as to the interactions between the customer and the provider. I would imagine inside this would be not just interaction with the service, but with management reports showing how the SLAs are meet. Cost analysis would be dealt with along side a good semantic interrupter to translate from one community of interest to another. But why a company should invest in a particular service or ecosystem of services seems to be missing. Beyond the technical reasoning, how does a company decide what should be kept and what should be leveraged.

One of my clients and I were talking about a particular service that they could either build, buy a license and host themselves or totally put it out in the cloud. His answer confirmed my thoughts. "I can't hire someone for that amount of money and get the service level agreement I need." However, another service was "too critical for the strategic advantage of this organization" to trust to another. Jack Welch quote in Straight from the Gut "Let your back-office be someone else's front office!" comes to mind. This becomes hard for IT managers to get since their jobs may be on the line. However, I believe that decision is the success difference maker and is part of what us who call ourselves enterprise architects should consider carefully.

As Brenda develops her research, advice and books (I hope) I believe she needs to consider with this model, some example cloud use cases to be able to test against. An example would be a sales tax generator that will calculate the appropriate sales tax for any geographic location in the world. Some situations that should be considered to make sure the model meets the needs are:

• Provider has all of the data, computing power, interfaces and sales force needed.
• Provider uses other providers for some of the resources needed but generally is a contained offering
• Provider only sells the services that others create
• Multiple providers sell the same service and compete on a variety of dimensions
• Multiple providers sell the same commodity service and compete mainly on cost per transaction
• Multiple providers sell the same service and dynamically providers come in to and out of the picture but the ecosystem has a static syntax and semantic definitions
• Multiple providers sell the same service and dynamically providers come in to and out of the picture but the ecosystem has a dynamic syntax and semantic definitions capability.

Again, I am ecstatic that a person of Brenda's caliber is taking on this challenge. I believe, slightly augmented, this model will serve her and her clients very well.

Bravo, my friend, Bravo! I, for one, am looking forwards to what comes next.

Events: The story of business

"There is no greater agony than bearing an untold story inside you. " - Dr. Maya Angelou

I read that quote and I think about Event Processing. Your probably thinking? "What are you talking about? What does a story have to do with event processing?"
It is this: The event is the captured story of business.

Event Processing is much more than the making clever observations

There are many people who focus on the mechanics of event processing. They think about mathematically how do we combine/aggregate/pattern recognize these event into something more meaningful. This is an incredibly important for without it, the "magic" is gone. But I feel there is a lack of people teaching about what is an event and how to we find them.

As said in many places, the event is the happening, the instant. Something happens to change the world as we know it. It might be something relatively small like a taking money out of an ATM or something with significant impact like a tsunami. But even small things change, in one context or another, the world as we know it. Event Processing has to do with detecting or deriving those changes in the world as we know it. The magic is not observing the happening, but capturing it in a way that we can react to it in our systems.

But how does one become a magician? Where to start? How do we bridge the gap between the physical domain and the virtual domain?

How do we change our mindset to see events

The main thing I would suggest is not to think about the individual events, but rather think of the story. We humans think in terms of stories. Of linear activities, of plot, of tension and inter-relationships. Is satisfying the customer order not a story? Or buying a stock at the right time? True, it might not even be interesting enough to make a "made for television" movie. But it is still a story. And in these stories are character and things happen to these characters. These characters do things and hand things off to other characters. Sure; you can be unromantic and call it a "business process" but still, it is a story. And in this story, as in all stories, there are natural times of interest where something of note happens. These times of interest are the events...

Think about your favorite movie or book. One of mine would be the Fellowship of the Ring by J.R.R. Tolkien. (perhaps you saw the movie) In this movie, some changes in the world were very obvious. Bilbo Left, Frodo got possession of the Ring, Nazgûl search for Ring, Frodo is stabbed with the Mordor Blade, Elrond heals Frodo to the best of his ability, etc. It is easy to see these are major changes in the state of the world. Of course there are more fine-grain events, but don't worry about them yet. Find the big ones in your story.
The way we find the big events in a movie is imagine someone says "Tell me what happened in the Fellowship of the Ring." You would tell all of, what the screenwriters call, the plot points. The major things. Again, there are different layers to the story. You can always go down a layer, later. But think of the large story plot points in your companies story.
"We get an order, we make a product, we bill the customer, we receive the check, we deposit the check" That, my friends, is a story. There are five events that occurred here. Then you start thinking of other end-to-end stories. "We discover a needed capability, we articulate that in a job description, we determine what we will pay for that capability, we advertise, we interview, we offer the position, we on-board the employee". Each of these plot points or events you should capture on a white board. Start to figure out when that event occurs, what do I know, who is involved, what products are being manipulated, when did it occur. These are the things that will make up the meta-data of your events.
Then ask, how can I capture that this occurred? Is this something that happens in a computer orchestrated process that I simply can publish that it happened? Is there a sensor that can sense the physical domain and determined it happened? Is there a way that I can determine it based on other events? Do I need to have a human put it into some UI because there are no other way, yet, to capture it? Does it come from customer, partner, information feed?
What is neat about these stories is that they are really the processes of our business. These events are the natural segmentation points in the process. So you can ask your question to go to the next layer: "What happens between receiving the check and depositing it?" This gives you more processes and more events, but more importantly, a natural hierarchy of events happens. It will also occur that you realize a story is part of a larger story. Of the ecosystems your company belongs; of the rivalry between you and your competitor. That things are occurring outside of your control that affects your story. These plot points are very important to capture.

How does this relate to Dr. Angelou's quote?
I was in a working group with Roy Schulte when an idea hit me. Events are happening in the bowls of the computer system. They generally happen inside of application code and are only observed by a small subset of applications. So, that program is the only one who can react to it. They are stuck there and are limited to expression by the program, which has been our practice, sending out command and control signals to order systems to do a particular part of the work. But that violates the principle of maximize cohesion and minimize coupling. Command and control couples the issuing system to the receiving system; increasing the complexity and ultimately the usefulness of the ecosystem. If the ecosystem had emotions, would it not be in agony?
With events, we decouple the applications. We give voice to the story outside of one application. We allow others to hear the events of the story and allow them to become another plot-line within the story without requiring the "main application" to become more complex. It allows other systems and capability to arise. To fill in gaps. To be built by others and work in tandem and be purposeful.

How does the story metaphor bring event processing to the masses?

We live and breath stories. Stories have as its foundation a state transition: An equilibrium, the equilibrium is lost because of some change (detect/derive), we trying to figure out how to get back to equilibrium (decide), we act (do), equilibrium is reestablished. When you think about your business processes as the stories of your business, the effort of noticing the large events and drilling down brings on a whole new way of thinking. The masses of business people and business analysts may not get what is happening under the covers to allow the derived events to be observed. But the story is a metaphor that all humans get!

So, tell your story well.

The 4Ds "Detect, Derive, Decide and Do"

The Interesting thing about the 4Ds

As I am sure all of us are familiar, it is in the framing of the problem that gives the greatest ability to abstract the problem.

Thus the framing where the 4Ds come into play.

Detect
This is the act of capturing the stimuli that comes from an outside source. This might be a sensor reading or a request to do some action.

The essence is moving it from one domain to another domain. In the case of temperature control, the heat sensor allows the capture of a stimuli of the "reality domain" to the "device domain". It may also be more virtual. I would argue that a web service call from one partner to another partner is the capture of the "request stimuli".

Included in this "area" is the conversion of the stimuli from the raw datum to a virtual version of that stimuli.

Note: You will see that I refer a lot to the stimuli. I do this because I believe that a service provider is constantly capturing stimuli in which to react. Perhaps even reacting to the lack of stimuli.

Derive
This is the act of correlating this stimuli with other stimuli or other derived understanding of the world. So what's does that mean?

When you think about situational awareness, it is basically an understanding of the world within the scope of context you care about. A big state machine in which reality (as it can be captured) is made virtual. Hmm.

CEP plays a large part in this because you can take various stimuli (called events and state changes), combine/aggregate/pattern recognize/intersect/unionize/... them and derive other stimuli (also called derived events). The power of which is events (happenings in a different domain) can be detected that can not be directly sensed in that other domain. Said differently, these events (or stimuli) that occur in a different domain can not be directly detected because there isn't a mechanism to capture the stimuli. However, by looking at patterns of events that you can detect in concert with other captured events, you are able to derive events in other domains that are not directly detectable . Wow! That's powerful stuff!

Let me give an example to illustrate this.

Let suppose that there exists (and there does) a camera that is able to take the picture of a car's license plate as it goes through a toll booth. This, obviously, would represent detect. This sensor takes a picture, converts the images of the plate and converts it to a variable that contains the letters and numbers of the license plate. As well, it includes the meta-data of that datum. In this case, the time and location that the picture was taken.

After some time, the car passed through another toll booth and another detect activity occurs. Except in this case, the meta-data of the license plate number datum has a different time and location. (I am assuming that the guy didn't just spin around the toll booth!)

Both of these events (stimuli) have been detect activities. Now comes the fun part.

Because the same entity owns both toll booths, it is able to compare the two detect events and derive a new piece of information. The elapsed time the motorist took between the two toll booths. This organization could then divided the distance (which they would know) by the elapsed time would would yield the average speed of the car associated with the license number between the two toll booths. This information would be added to meta-data about the trip which is a higher order abstraction that the passing of a toll booth. ( I am assuming that this isn't James Bond who has license plates that can swivel to a new one )

So, the adding of information or determining situational awareness is the role of the derive. I want to derive more information about the trip the license plate took.

More specifically, the determination that the motorist was speeding (specifically violating the law that regulates speed of motorists along that road) would also be derived.

Decide
The decide section, as its name suggests, is determining if and what to do because of the awareness of the situation.

In our example, the rules (decision logic) may specify that if the speed was over a certain tolerance, then we issue a ticket, if it is less than that tolerance, we ignore the stimuli.

Business rules engines are very good at handling this capability in a very configurable manner . But just as likely a case statement or IF THEN ELSE might represent the decide action.

Do
Again, being the master of the obvious, the name of this section represent doing the activity that was decided to be done.

In the case of the traffic example, it might issue a request to the ticketing web service to cite the offending motorist or that capability might be within the boundaries of the service and would execute its own mechanism.

How the 4Ds solves the Problem
Now that the 4Ds are understood and the offending motorist has contributed to the city's budget, your probably wondering, so how does this help me develop a solution?

With all abstractions, we are looking for ways to divide the problem. But not just for divide and conquer purposes. There are different mindsets and concerns that each of these areas would require. And potentially different expertise. A person that knows how to manipulate the sensor to take the readings dealing with harmonics and weight levels for a bridge may not necessary know the statistical equations to put those data points together to determine the bridge's stress or decay. Similarly that individual or team may not decide how we react to those stress levels. Nor are they necessarily the ones who will put the mechanics of put into the action the decisions. Perhaps they are, but this solution allows these factors to be separated.

Does this go beyond Event Processing and SOA
As I said earlier, I don't believe this is limited to simply CEP or expert systems. This is the nature of a reactive system, which all computer systems in truth are. Even "proactive" systems are reacting to an understand of the world around us and a potential threat or opportunity (also known as situational awareness). The difference, in my mind, between a proactive and a reactive system is a proactive system is one where the actions occur before that HAVE to be or for marketing purpose before the competition reacts to it.


Summary
So, in summary, I hope that I have given you reason to 1) think about systems as reactions to stimuli (events (changes in the world around us)) and 2) that when we break the solution up into the 4Ds, we are dividing the solution into logical segments that takes advantage of the discrete types of thinking.

Notes:
As always, I want to have a humble ego when it comes to my ideas. I realize that no man or idea is an island unto itself, but rather stands upon the shoulders of giants. I hope that if anything, it acts as a springboard for others to have even better ideas or acts to help enhance or dismiss my ideas into something more accurate and truthful. So, I will take any comments as constructive criticism.

EA without Vision is dead

This is response to an interesting post from Richard Veryard of the Next Practice Research Initiative. Two people I have high regard Alex Buterman and Brenda Michelson commented on it, so I figured it was worth reading. Hopefully this post will be worth reading too. Warning, this was written @ 1AM after watching House, so it might be a bit….1AMish (as in like something written at 1 AM not the religious order who practice living simply in a complex world)

The problem is there isn't a good way of doing Enterprise Architecture (or anything structural thinking aid) without having someone with vision and power. EA is a structure designed to hold information about a business and make it accessible as well as the processes, policies, etc. Blah blah blah. As the Burton Group's Report points out, there is value in being results oriented and aligning yourself to the business and learning to communicate. (the Burton's Group Report was the subject of what the post by Richard Veryard which was commented by Alex and Brenda which lead me to post this! So, its kinda germane)

But what good is that?

I am becoming more and more convinced that EA, and most other IT initiatives fail not because they aren't understood or don't have the management backing or they weren't completed enough. No, I believe the reason is the people involved don't have vision. Most people are not innovative nor do they have vision. (Although they do have perspective!) They regurgitate what others have tried without understanding the physics behind why what the mimicked did worked. If you don't get that sentence. Stop reading!

What seems to be missing in our search for silver bullets and hyper-hyped philosophies is the fundamental principle that all of them (the hyped philosophies, methodologies, do-hickies) are just tools. Things! Things to help us go! But if they that wish to go are without a person who understand where to go inside of their gut, the tools are at best ok.

Take a simple example of SOA. We debate endlessly as to the correct form of SOA. how large, how generic, how encompassing should it be? Should processes be part of a service, consumed by a service or both? These questions are irrelevant, yet that is what people's minds are able to grab a hold. They are stuck in metaphor land. Thinking in terms of the metaphor rather than terms of reality. It's like all those episodes of Star Trek where they said "It's like taking a match to stick of dynamite laced with gasoline." The problem isn't the metaphor, it is the people who believe they understand warp theory now because they know what happens when you take a match to a stick of dynamite laced with gasoline.

Enterprise Architecture, in the hands of a virtuoso, is wonderful because order can be established inside of the processes, activities and stuff that a enterprise does. But the goal isn't order and establishment (well it is for GE and Six-Sigma places who want to take variation out of their processes but that is beside the point) or even to create flexibility and agility. To quasi-quote the "Good Book" (a.k.a. Bible (the Christian one, not the SQL one)) If I have the flexibility of ballerina and have not vision, then I am a nimble dancer in the wrong act of the play. If I have agility and can change the direction of my company on the slightest change of the market, but have not vision, my agility is for not because I will sit and spin with no direction like a child on a sit and spin and spinning until they puke! (Not sure how that last quote ended up there!)

All of our programs, architectures, frameworks, etc. bring order in the chaos. But unless you have someone who pushes you into the chaos your stuck with status-quo.

Brenda has been tweeting about "is a larger or smaller consulting firm better for helping a company 'do' EA (or fill in the blank)" I say, it doesn't matter. What does matter is to have a person of vision to lead us not just in the safe trusted areas, but into the place where Angels fear to tread for there is way to … (Add in your favorite hyper-hyped business term here)

Last thing, If you call yourself an Enterprise Architect, you should buy (that way he gets a royalty) and read Stephen Pinker's book "The Stuff of Thought: Language as a Window into Human Nature." It has a great chapter called "The Metaphor Metaphor." After you read that chapter, read "Down the rabbit hole." If you don't like either of those, he has a great chapter on explicative usage.

The last section of one of the chapters (I think it was "down the rabbit hole") he talks about Plato's cave. Where people are chained head and body made to look at the back wall. There is a fire behind them and people are using cut-outs to make shadows on the walls. The people think "This is reality!" for this is all they know. One of the people escape and find out that they are actually in a cave and real reality is much better. He comes back into the cave and tries to get his brothers out so that they can see the real reality. But they refuse believing instead that what he saw was captured in what they saw on the wall. If you call yourself an enterprise architect, (and actually read this nonsense this far) then you need to be that person who chains are off and can see outside the cave. Use the tools that you have in EA and others frameworks to pry your brothers (or sisters) eyes away from the back of the wall.