Events:The Great De-coupler - Determine the Independance score of an event system.

Events are the great de-coupler because their very nature changes our prospective from command-oriented computing to situation-oriented.

Think about it. Most of the things we have learned in CIS courses was how to help the computer issue commands. It is easier, in some senses, to just command the reaction that we want rather than put it in context so that the situation can be understood by people who didn't write the event publisher. When we simply command the reaction we want, we are increasing coupling. We are tying a dependency from the application that raised the event to the application that reacts to it. I contend that by properly creating the event object with the appropriate references and ontology that support those references that we can decouple our events and our applications even further! We can even go to the point of reacting to events that are created outside of our sphere of influence automatically.

A goal of the event processing community should be to make semantically-aware events that can be consumed by an application or engine with no knowledge of or organizational affiliation to the event's publisher.

To start off, we need to design events to live outside of a particular application, department and organization and even an particular industry. We have already taken strides when various taxonomies and ontology for industry are created. They take the real-world nouns, processes, and state-changes of the industry and give it a namespace so that all may call things the same. This trend promises to continue and become better at defining them.

The table below defines the various levels of independence for an event object. This isn't just about how do we create a good event object that the system can use. This is about building an ecosystem of information that surrounds the set of events for a given organization giving it a rich environment to influence.

For reference sake:

• Event: The change of state of a noun.
• Event Object: A container that encapsulates the meta-data associated with an observation of an event

Category	Description	Comment
Decoupled	The event object [EO] must exist independent of any computing algorithm that would operate upon it's content. Must be independent of reactions to it.	This leaves algorithm that manipulates the container, moving it, etc. This doesn't mean that things can't operate on its content, just that the event object doesn't rely on the operation for its existence. (Obvious exception is the event creation algorithm)
Subscribe-able	A mechanism must exist so that a subscription of the event object can be requested and the event object delivered upon publication.	This is the classic event-driven architecture.
Descriptive not prescriptive	The event object must indicate what happened (state change/action/lack of state change) and not indicate how to react to what happened. An event object must be independent of commands	This one separates out awareness from commands meaning that an event object should describe the situation not issue a command. There are a lot of messages that are decoupled and subscribe-able but the message issues a command or issues constraints to the reaction to a notification. These would not satisfy this level of events.
Contextual	EO must indicate/identify the noun (object) the event is in reference to EO must indicate the state that the noun changed to EO may indicate the state of the noun that the noun changed from An event object must be independent of embedded knowledge in the ecosystem.	This basically says that an event must self-containing it's direct references. The Event Object should answer the questions: What happened? To whom? When? Where? What are the details? What is the certainty?... All of these are referenced inside the event object.
Referenced	EO references must be governed by a taxonomy/ontology An event object must be independent of semantic ambiguity	Referenced means that all pieces of context are referenced to some source outside the event. Similar to Fielding's REST concepts. It should be a URI or something that can identify the subject.
Dimensional	Nouns have defined dimensions that are made up of defined states or values. Since events are an observation of the change of state for a noun, the referenced taxonomy should have a well defined dimensions, states, and values for all nouns under its jurisdiction.	In my view, nouns have dimensions (attributes) that either describe a state or a property. In this maturity, the dimensions must be defined as well as what legal state/value can exist in that dimension.
Cross-referenceable	The taxonomies must be systemically translatable to other taxonomies causing semantic interdependence.	By developing cross-referenced taxonomies, organization outside of the sphere of influence of the event publisher can understand and utilize the events without requiring human intervention.

Cloud-o-gram

When I first saw Brenda Michelson's post Cloud-o-gram, I couldn't help think about sneaking downstairs after my parents went to sleep to watching Saturday Night Live. One episode they had was this land-shark that would try to get into people's houses or apartments by pretending to be a candy-gram delivery. But unlike the delivery of the most cleverest of all sharks, this cloud-o-gram reference architecture seems to be the real deal.

One of the fundamental problems facing things that allow ecosystems to develop on it is how do you segment it into areas so that you can focus your attention? When you do, what lines do you draw so that your segmentation isn't arbitrary? When I look at the cloud and start to talk about it to people (and sometimes just to myself) I fluctuate between customer POV, Provider POV, Ecosystem POV, Machine POV and land-shark POV. And when you add recursion that a provider of a service is really reselling a service that is consuming yet another service which that same physical service might actually be sold by yet another provider! Oy Vey!

My read is that Brenda has nicely separated out what is sold (offerings) from the mechanisms that provide what is sold (cloud computing environment) from the sales management of what is being sold (For the/About the cloud) from the agreement to sell (SLA). Genius!

This reference architecture will be the organizing mindset behind the Elemental Link's research (and I suspect a book or two). What I am unsure of at this point is does the reference architecture allow for multiple points of view? As with most services, the veil of encapsulation allows the consumer to "ignore the man behind the curtain", but to their own detriment in many cases. The power of abstraction hides complexity (as it should) but also increases fear of the unknown or un-understood. This is why what Brenda is offering is important for consumers and providers alike. To dispel the magic to allow rational decisions to be made.

As much as I am glad that she is taking on this important work, I feel that there is a significant area missing: customer reasoning. As with most services, this model also has a boundary as to the interactions between the customer and the provider. I would imagine inside this would be not just interaction with the service, but with management reports showing how the SLAs are meet. Cost analysis would be dealt with along side a good semantic interrupter to translate from one community of interest to another. But why a company should invest in a particular service or ecosystem of services seems to be missing. Beyond the technical reasoning, how does a company decide what should be kept and what should be leveraged.

One of my clients and I were talking about a particular service that they could either build, buy a license and host themselves or totally put it out in the cloud. His answer confirmed my thoughts. "I can't hire someone for that amount of money and get the service level agreement I need." However, another service was "too critical for the strategic advantage of this organization" to trust to another. Jack Welch quote in Straight from the Gut "Let your back-office be someone else's front office!" comes to mind. This becomes hard for IT managers to get since their jobs may be on the line. However, I believe that decision is the success difference maker and is part of what us who call ourselves enterprise architects should consider carefully.

As Brenda develops her research, advice and books (I hope) I believe she needs to consider with this model, some example cloud use cases to be able to test against. An example would be a sales tax generator that will calculate the appropriate sales tax for any geographic location in the world. Some situations that should be considered to make sure the model meets the needs are:

• Provider has all of the data, computing power, interfaces and sales force needed.
• Provider uses other providers for some of the resources needed but generally is a contained offering
• Provider only sells the services that others create
• Multiple providers sell the same service and compete on a variety of dimensions
• Multiple providers sell the same commodity service and compete mainly on cost per transaction
• Multiple providers sell the same service and dynamically providers come in to and out of the picture but the ecosystem has a static syntax and semantic definitions
• Multiple providers sell the same service and dynamically providers come in to and out of the picture but the ecosystem has a dynamic syntax and semantic definitions capability.

Again, I am ecstatic that a person of Brenda's caliber is taking on this challenge. I believe, slightly augmented, this model will serve her and her clients very well.

Bravo, my friend, Bravo! I, for one, am looking forwards to what comes next.

The 4Ds "Detect, Derive, Decide and Do"

The Interesting thing about the 4Ds

As I am sure all of us are familiar, it is in the framing of the problem that gives the greatest ability to abstract the problem.

Thus the framing where the 4Ds come into play.

Detect
This is the act of capturing the stimuli that comes from an outside source. This might be a sensor reading or a request to do some action.

The essence is moving it from one domain to another domain. In the case of temperature control, the heat sensor allows the capture of a stimuli of the "reality domain" to the "device domain". It may also be more virtual. I would argue that a web service call from one partner to another partner is the capture of the "request stimuli".

Included in this "area" is the conversion of the stimuli from the raw datum to a virtual version of that stimuli.

Note: You will see that I refer a lot to the stimuli. I do this because I believe that a service provider is constantly capturing stimuli in which to react. Perhaps even reacting to the lack of stimuli.

Derive
This is the act of correlating this stimuli with other stimuli or other derived understanding of the world. So what's does that mean?

When you think about situational awareness, it is basically an understanding of the world within the scope of context you care about. A big state machine in which reality (as it can be captured) is made virtual. Hmm.

CEP plays a large part in this because you can take various stimuli (called events and state changes), combine/aggregate/pattern recognize/intersect/unionize/... them and derive other stimuli (also called derived events). The power of which is events (happenings in a different domain) can be detected that can not be directly sensed in that other domain. Said differently, these events (or stimuli) that occur in a different domain can not be directly detected because there isn't a mechanism to capture the stimuli. However, by looking at patterns of events that you can detect in concert with other captured events, you are able to derive events in other domains that are not directly detectable . Wow! That's powerful stuff!

Let me give an example to illustrate this.

Let suppose that there exists (and there does) a camera that is able to take the picture of a car's license plate as it goes through a toll booth. This, obviously, would represent detect. This sensor takes a picture, converts the images of the plate and converts it to a variable that contains the letters and numbers of the license plate. As well, it includes the meta-data of that datum. In this case, the time and location that the picture was taken.

After some time, the car passed through another toll booth and another detect activity occurs. Except in this case, the meta-data of the license plate number datum has a different time and location. (I am assuming that the guy didn't just spin around the toll booth!)

Both of these events (stimuli) have been detect activities. Now comes the fun part.

Because the same entity owns both toll booths, it is able to compare the two detect events and derive a new piece of information. The elapsed time the motorist took between the two toll booths. This organization could then divided the distance (which they would know) by the elapsed time would would yield the average speed of the car associated with the license number between the two toll booths. This information would be added to meta-data about the trip which is a higher order abstraction that the passing of a toll booth. ( I am assuming that this isn't James Bond who has license plates that can swivel to a new one )

So, the adding of information or determining situational awareness is the role of the derive. I want to derive more information about the trip the license plate took.

More specifically, the determination that the motorist was speeding (specifically violating the law that regulates speed of motorists along that road) would also be derived.

Decide
The decide section, as its name suggests, is determining if and what to do because of the awareness of the situation.

In our example, the rules (decision logic) may specify that if the speed was over a certain tolerance, then we issue a ticket, if it is less than that tolerance, we ignore the stimuli.

Business rules engines are very good at handling this capability in a very configurable manner . But just as likely a case statement or IF THEN ELSE might represent the decide action.

Do
Again, being the master of the obvious, the name of this section represent doing the activity that was decided to be done.

In the case of the traffic example, it might issue a request to the ticketing web service to cite the offending motorist or that capability might be within the boundaries of the service and would execute its own mechanism.

How the 4Ds solves the Problem
Now that the 4Ds are understood and the offending motorist has contributed to the city's budget, your probably wondering, so how does this help me develop a solution?

With all abstractions, we are looking for ways to divide the problem. But not just for divide and conquer purposes. There are different mindsets and concerns that each of these areas would require. And potentially different expertise. A person that knows how to manipulate the sensor to take the readings dealing with harmonics and weight levels for a bridge may not necessary know the statistical equations to put those data points together to determine the bridge's stress or decay. Similarly that individual or team may not decide how we react to those stress levels. Nor are they necessarily the ones who will put the mechanics of put into the action the decisions. Perhaps they are, but this solution allows these factors to be separated.

Does this go beyond Event Processing and SOA
As I said earlier, I don't believe this is limited to simply CEP or expert systems. This is the nature of a reactive system, which all computer systems in truth are. Even "proactive" systems are reacting to an understand of the world around us and a potential threat or opportunity (also known as situational awareness). The difference, in my mind, between a proactive and a reactive system is a proactive system is one where the actions occur before that HAVE to be or for marketing purpose before the competition reacts to it.


Summary
So, in summary, I hope that I have given you reason to 1) think about systems as reactions to stimuli (events (changes in the world around us)) and 2) that when we break the solution up into the 4Ds, we are dividing the solution into logical segments that takes advantage of the discrete types of thinking.

Notes:
As always, I want to have a humble ego when it comes to my ideas. I realize that no man or idea is an island unto itself, but rather stands upon the shoulders of giants. I hope that if anything, it acts as a springboard for others to have even better ideas or acts to help enhance or dismiss my ideas into something more accurate and truthful. So, I will take any comments as constructive criticism.

EA without Vision is dead

This is response to an interesting post from Richard Veryard of the Next Practice Research Initiative. Two people I have high regard Alex Buterman and Brenda Michelson commented on it, so I figured it was worth reading. Hopefully this post will be worth reading too. Warning, this was written @ 1AM after watching House, so it might be a bit….1AMish (as in like something written at 1 AM not the religious order who practice living simply in a complex world)

The problem is there isn't a good way of doing Enterprise Architecture (or anything structural thinking aid) without having someone with vision and power. EA is a structure designed to hold information about a business and make it accessible as well as the processes, policies, etc. Blah blah blah. As the Burton Group's Report points out, there is value in being results oriented and aligning yourself to the business and learning to communicate. (the Burton's Group Report was the subject of what the post by Richard Veryard which was commented by Alex and Brenda which lead me to post this! So, its kinda germane)

But what good is that?

I am becoming more and more convinced that EA, and most other IT initiatives fail not because they aren't understood or don't have the management backing or they weren't completed enough. No, I believe the reason is the people involved don't have vision. Most people are not innovative nor do they have vision. (Although they do have perspective!) They regurgitate what others have tried without understanding the physics behind why what the mimicked did worked. If you don't get that sentence. Stop reading!

What seems to be missing in our search for silver bullets and hyper-hyped philosophies is the fundamental principle that all of them (the hyped philosophies, methodologies, do-hickies) are just tools. Things! Things to help us go! But if they that wish to go are without a person who understand where to go inside of their gut, the tools are at best ok.

Take a simple example of SOA. We debate endlessly as to the correct form of SOA. how large, how generic, how encompassing should it be? Should processes be part of a service, consumed by a service or both? These questions are irrelevant, yet that is what people's minds are able to grab a hold. They are stuck in metaphor land. Thinking in terms of the metaphor rather than terms of reality. It's like all those episodes of Star Trek where they said "It's like taking a match to stick of dynamite laced with gasoline." The problem isn't the metaphor, it is the people who believe they understand warp theory now because they know what happens when you take a match to a stick of dynamite laced with gasoline.

Enterprise Architecture, in the hands of a virtuoso, is wonderful because order can be established inside of the processes, activities and stuff that a enterprise does. But the goal isn't order and establishment (well it is for GE and Six-Sigma places who want to take variation out of their processes but that is beside the point) or even to create flexibility and agility. To quasi-quote the "Good Book" (a.k.a. Bible (the Christian one, not the SQL one)) If I have the flexibility of ballerina and have not vision, then I am a nimble dancer in the wrong act of the play. If I have agility and can change the direction of my company on the slightest change of the market, but have not vision, my agility is for not because I will sit and spin with no direction like a child on a sit and spin and spinning until they puke! (Not sure how that last quote ended up there!)

All of our programs, architectures, frameworks, etc. bring order in the chaos. But unless you have someone who pushes you into the chaos your stuck with status-quo.

Brenda has been tweeting about "is a larger or smaller consulting firm better for helping a company 'do' EA (or fill in the blank)" I say, it doesn't matter. What does matter is to have a person of vision to lead us not just in the safe trusted areas, but into the place where Angels fear to tread for there is way to … (Add in your favorite hyper-hyped business term here)

Last thing, If you call yourself an Enterprise Architect, you should buy (that way he gets a royalty) and read Stephen Pinker's book "The Stuff of Thought: Language as a Window into Human Nature." It has a great chapter called "The Metaphor Metaphor." After you read that chapter, read "Down the rabbit hole." If you don't like either of those, he has a great chapter on explicative usage.

The last section of one of the chapters (I think it was "down the rabbit hole") he talks about Plato's cave. Where people are chained head and body made to look at the back wall. There is a fire behind them and people are using cut-outs to make shadows on the walls. The people think "This is reality!" for this is all they know. One of the people escape and find out that they are actually in a cave and real reality is much better. He comes back into the cave and tries to get his brothers out so that they can see the real reality. But they refuse believing instead that what he saw was captured in what they saw on the wall. If you call yourself an enterprise architect, (and actually read this nonsense this far) then you need to be that person who chains are off and can see outside the cave. Use the tools that you have in EA and others frameworks to pry your brothers (or sisters) eyes away from the back of the wall.

SOB - Service Oriented Business

I was having lunch with a friend and he asked me: "What is a simple definition for SOA that I can tell people when they ask?"  I thought about that and all the answers I came up centered around "How do we build it."  

When I googled it, I got lots of answers, but they were very technical in nature or used service in the definition:

Oasis: A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations. citation

Microsoft: 

Service orientation is an approach to organizing distributed IT resources into an integrated solution that breaks down information silos and maximizes business agility. Service orientation modularizes IT resources, creating loosely coupled business processes that integrate information across business systems. Each IT resource, whether an application, system or trading partner, can be accessed as a service. These capabilities are available through interfaces; complexity arises when service providers differ in their operating system or communication protocols, resulting in inoperability.

Citation

IBM: Service Oriented Architecture (SOA) is a business-centric IT architectural approach that supports integrating your business as linked, repeatable business tasks, or services. 

Gartner: An application topology in which the business logic of the application is organized in modules (services) with clear identity, purpose and programmatic-access interfaces. Services behave as "black boxes": Their internal design is independent of the nature and purpose of the requestor. In SOA, data and business logic are encapsulated in modular business components with documented interfaces. This clarifies design and facilitates incremental development and future extensions. An SOA application can also be integrated with heterogeneous, external legacy and purchased applications more easily than a monolithic, non-SOA application can. Citation

None of these definitions were something I could memorize and regurgitate when a vice-president asked "What's this SOA thing?"

My definition is simple: "SOA is a way of organizing work that maximizes the consumer / provider metaphor"  A friend suggested I replace "metaphor" with "relationship."

As I was thinking about this definition, I was studying agile programming and there was a reference to the type of company: Functional-oriented vs. project oriented were what the book was referring.  My brain said, "What about service-oriented?"

Can we make a service oriented business (SOB)?  Certainly we have thought of that acronym about employees of a business who believes incorrectly that they are service-oriented.  But can we actually make a business that is completely service-oriented in the consumer / provider metaphor?  What would such a business look like?

Obviously at the top are services that are provided to other entities which they (asuumeingly) pay to consume.  Internal to these services, some work is accomplished.  But probably there are also underpinning services as well.  Services that we rely to make our service perform, stocked, etc.  And those services have underpinning services and so forth.

But if we consider SOA as maximizing the consumer / provider relationship, we could consider the underpinning service as its own business.  A business that "sells" its own products/services.  As a network of these arise, we can see the value matriculating from the end-consumer through our network of inter-related services.

What I find interesting about this concept is that it is value based.  A service invests its income in providing its customers what they want.  I know we have hear the "treating the other employees as customers" rhetoric hundreds of times.  But if we truly created a business as a network of services, would that change?

One other interesting thought that I have about this network of services.  It is ripe for outsourcing.  If we divided a company into logical semantically accurate functionality [work] and consider the "service" as an internal provider of that work and then we found out an external party can offer that same service at the higher value / cost matrix.  Wouldn't that inspire better investment into that service internally?

Said a different way.  Couldn't this help us meet our goal of All work that is necessary for a business to operate should be provided by an entity whose core competency is doing that particular work.

Just some things to think about.

Other types of functionality... where does it go on the cloud

I have been curious of how will the cloud evolve.  We have picked basic functionality to start the cloud.  Queuing, Storage, instruction processing, network are the blocks that the rest of IT is built upon.  As an industry, those blocks have been/are being solidified.  What I would like to know is what is next to be built or perhaps next after that.  As usual, I have an opinion, but it will, also as usual, require a bit of context.  I hope the conclusion will be worth it!

My contention is this: "The cloud" is a mechanism to enable services of commodity functionality  [Which is another name for work] and as such, a well understood taxonomy or organization of these commodity functionality will need to be developed in order for higher level functionality to prosper in the cloud.  

To understand this, understand that functionality can be described as: Core (value adding), Unique (necessary, but no provider we trust), Commodity (necessary and providers we trust)  and Extraneous (not necessary).  Core competencies (functionality), as described by Jack Welch, is work that a company focuses on being the best at to set itself apart from its competitors.  Unique is work that is needed to be done but it doesn't set us apart directly and there aren't providers that we can trust to do the work.  Commodity is work that is needed to be done that doesn't provide differentiation but there are providers who can do the work.  Extraneous are waste in the lean six-sigma sense.

In real-world products, we have seen business outsource accounting, payroll, manufacturing, sales, IT.  Following Jack Welch's advice of focus on core competencies and allow others to offer a service which encapsulates the work that isn't core to your value proposition.  This makes sense. It leads a company to only, for the most part, focus on things that matter to its bottom line.  However, a company will still need to do unique functionality because no one else can.  Our goal should be to convert, as much as advisable in terms of security, functionality from unique to commodity.  Even potentially creating new industries to provide that functionality.  Stated differently all work that is necessary for a business to operate should be provided by an entity whose core competency is doing that particular work.

So applying this to information technology, there are algorithms that are core, unique, commodity and waste.  If we follow the same advice, our efforts should be focused on our IT's core competency.  Work that is unique we should plan how to make it commoditized.  Commodity should be being done by others and waste should be eliminated.  But how do we make this happen?  

With the advent of services and well-understood interfaces, it has become operationally easier to separate out the core from the commodity services.  The cloud starting from the ground up, with storage, database, computing power, networking.  Functionality that all systems share and because they have stable, mature, well-understood interfaces.  This is not really due to the efforts of the cloud community for they stood on the shoulder of giants. Rather it was the efforts of IT Vendors and standards groups that standardized the interface allowing the creation of their products.  Remember these standards allowed the syntactic ambiguity and also, to the extent possible, the semantic ambiguity to disappear.  However, this canonization effort is expensive, time-consuming and potentially a pre-ripe standard may stifled innovation to soon.

High level functionality such as business rules, calculation, decision logic may indeed be capable of being a commodity.  However, trust, the ecosystem of providers, the organization of functionality and the interfaces standards for the functionality are immature.  Thus it traps potential commodity functionality as unique.  Or it requires that business purchases a large package that stands in for the undeveloped ecosystem which can hinder your IT organization from being enabling business differentiation which should be its primary objective.

SAP and other large package vendors will tell you that they are opening up to allow highly customized processes and manipulation of data.  They will even allow you to call a web service from an external (to SAP) provider (probably will be you!) and use the response in your process.  So, in essence SAP (and all the other ones too) has turned into a platform that provides a rich array of functionality and process to accelerate your mapping of your emulated business to your real business.  This platform can become the focal point of your business.  Is this new?  no.  Is this bad? no.  But the ecosystem needs to extend out.  Potentially even replacing parts of the rich array of functionality or perhaps replacing the platform.  A business using a package should ask itself, what is the core competency of our large package software and map that answer to the previous goal: all work that is necessary for a business to operate should be provided by an entity whose core competency is doing that particular work.  No vendor can provide all functionality to all types of parties as part of their core competency.  At least not to the determent of some of the functionality.  So, how do we build an ecosystem that will allow best of breed augmentation using external functionality?  How will we know who has what?  And how difficult will it be to have multiple providers or to switch providers?  Asked another way, how do we abstract out the functionality from the provider?

These, I believe, are the core questions to prompting us to make high order functionality available on the cloud.  We tried UDDI, but there wasn't a good taxonomy developed to describe where the functionality fit.  As well, there wasn't a good semantic ontology developed to describe the interfaces.  Nothing helped us be agnostic to the physical interface.

So my solution:

Imagine in a folks-onomy way a provider builds a version of their functional taxonomy.  This would include the semantics of what the functionality does.  As well, the semantics of the interface.

This world would also includes a more generic taxonomy built with help of linguists, computer scientists that is semantically accurate.  These taxonomies will be built, probably, along industry segments lines.  The company accomplishing this intellectual property, will work with individual providers to help them map their own taxonomy to the more generic taxonomies potentially across industry boundaries.  Where holes appear, the company will modify the generic taxonomies to make it a more complete and living entity.

The beauty of this is when a company wants to consume commodity functionality.  They search the functional taxonomy to find what they need.  Their platform/package provider will jump start their own taxonomy and the individual consumer will modify it to match their particular implementation.  The individual company will a more specialized functionality for its business than its package provider provides.  Because providers publish what functionality they provide and a match can be found at run-time.  Also, because the interface is semantically tagged, the consumer can provide the correct information based on its data's association with the generic taxonomy at run-time and convert the response to its own format.  

Therefore, the physical interface doesn't need to be well-understood, just semantically understood.  Which has been a huge problem all along.  We don't semantically tag our information (as a general rule) to an authoritative source .  In the same vein, we could do the same for describing nouns, processes and events.  It is in organizing our functionality (work) and our information that will allow our trapped unique functionality to become commoditized and therefore meet our goal.  

I think a company that can build and maintain the taxonomy so that semantic accuracy can be a first order principle in software design would be an awesome place to work.  As the cloud grows and encapsulates higher order functionality, semantics will be foremost!  Well, until it itself is commoditized.  

Anyone want to help me build this company?

The Emulated World

The work the glossary group (of which I am a member) of the event processing technical society is incredibly interesting.  How do you take all of the terms of a new, upcoming transformation in the way we design systems and document them in a way as to expand the usefulness and usability of the concept is what we are trying to accomplish.  To do so there are at least a couple of areas: 1) Philosophy, 2) Descriptive/Proscriptive and 3) Mechanics.  (Probably more!)

I want to write a bit about the philosophy behind events and to a certain extent computer science in general.  My contention is that what we are building with computer systems in general is an emulated world that allow us in a query-able fashion have/get/develop an understanding of the real world.

An examples to make this more clear.  

A warehouse has many objects in it.  Each set of objects (Say a pallet) has a location, quantity, item type, an owner and potential endless attributes.  In the real world, you can't stand in the middle of an isle and say "How many widget Xs do we have?" and expect an answer.  You would have to count them.  So, we develop an inventory system that keeps track of actions done to all of the sets of objects: Move, ship, receive, destroy, assemble, disassemble, etc.  The inventory system is emulating reality and allowing the events that occur in the real world to stimulate process in the emulated world that alters the persistence of this information.

So, the problem becomes how do we get the events in the real world into the emulated world?

Lets keep with the previous example.  An early ERP system would have created an order for work to be done (The move, destroy, ship, receive, assemble, disassemble events), some entity would accomplish that action and mark that it was done.  Usually this all happened on a piece of paper called a pick list.  The pick list would be turned in to a inventory clerk who would enter the new information to the inventory system and it would be updated.  This caused problems because of latency of information update, accuracy.  So people got smarter and they barcoded it so that when the forklift operator dropped it off, it would be scanned and updated quicker.  Now we attach RFID transmitters on these items so that they can be tracked with finer granularity and all movements can be even easier to capture.

This is what I call crossing the threshold between the real and emulated worlds.  We have a variety of means of getting information to pass over the threshold.  User or systemic interfaces, sensors and actuators are all means of getting these events across the border.

But what is our goal in all of this?  We want to have as accurate as possible representation of what is happening in the real world in this emulated world.  Why?  Because the emulated is what we can manipulate, model and make sense of what is happening in the very complex real world.

So, what does all of this have to do with CEP?

There are two types of events.  Those we can sense (without intervention) their occurrence in the real world and therefore react-able in the emulate world and those we can't.  The number of events that can be sensed via sensors such as RFID, biometrics, thermometers, red-light cameras are increasing daily.  A lot of research and money are being spent on increasing the ability of the emulated world to "see/sense" into the real world.

But what about those events we can't sense yet?  Traditionally,  we have built user interfaces to allow humans to intervene and "enter" the event into the emulated world.  The problem with this method is the reliance on a person to intervene?  How accurate will that be?  How timely?

Complex event processing allows a different mechanism to determine that event occurred in the real world.  We can compute it.  We derive an event that we can't directly sense through the combination of events that we can sense.  As an example, we can't directly measure that a car is speeding. (Although that is probably coming)  The speed event however has implications to events that capture-able.  The most obvious is the radar gun.  A police officer uses a radar gun, determines the motorist is speeding, writes a ticket, that ticket is entered into the system updating the emulated world.  Another method would be to have high-speed camera at certain intervals that take pictures of motorists license plates noting the car (identified by the license) and the time and the place.  Further cameras would record the same car at a different time and different place.  The math would be easy to determine how long it took to travel from point A to point B and if that would be in violation of the traffic laws.

Regardless if we have the technology to implement this particular scenario isn't important.  But it illustrates how an event is not an island.  More to the point there are potentially many ways of determining if this event occurred.  Some of them would be able to pass through the threshold between the real and emulated worlds allowing computation to determine the realization of the more interesting event.

Next post will be about how do we name events.

Evangelist

I was talking with a friend of mine today and he was making fun of the fact that I call myself an SOA Evangelist. He said there are some interesting anagrams for evangelist, so I had to go see. Here are some interesting ones that came up for SOA Evangelist:

Galvanise Toes
Navigates Loses
Nostalgia Eves
Salvation Gees
Envisage Altos
Negative Lasso
Gasolene Vista
Naivetes Goals
Loveseat Gains
Eating Salvoes (Cheers)
A Evangelist So (should be a question... A evangelist, so?)
A Negative Loss
A Navel Egoists
A Goat Evilness
Again Lose Vest

In particular I think the Naivetes Goal and Negative Lasso are particularly interesting.

Intersection between Enterprise Architcture, ITIL and the SOA Movement

The past few weeks I have had an ephitheny about the relationship between ITIL and Enterprise Architecture. The relationships that the Enterprise Architecture bricks describe the information needed for a configuration item (CI) inside of the CMDB. That a CI has to have a type associated with it and that type should be a brick inside of EA.

How I came about this discovery is I was trying to figure out what are the CI for our application infrastructure. It seem to me that ITIL focused on the component aspect of things inside of the production environment. This usually consisted of things like physical devices and brought-in software. But the more I thought about that, it seemed as if something was missing. Bespoke (British for custom-made) software (stuff you build yourself) didn't seem to be as well covered.

So, it seemed as if the connection between the service provided and the hardware, software used were connected through some indirect way. However accurate, it seem something was missing.

Dr. Warner Vogel, CTO of Amazon, gave a speech on Enterprise Architecture where he indicated that Amazon was really a platform where buyers and sellers get together. The largest seller using the Amazon Platform was the Amazon Store. But when you think of Amazon, you have to think of the various domains that are interconnected. One domain is the platform which does things like provides services such as item management, buyer-centric human and systemic interfaces, seller-centric human and systemic interfaces, shopping cart management, promotions, advertiser-centric human and systemic interface, etc.

But there are other domains involved in the ecosystem. Front-end providers which sell through their interfaces items in the Amazon product set. Also, you have product providers (sellers) which offer their products and services as part of the Amazon product set. Another domain is the actual buyer who has their own process of purchasing a product that may connect to any of these sellers.

The point is that there are many intersections between these domains. That became significant to me in my quest for understanding. I asked myself, "What is really happening at these intersections." The answer I got was that control was being transferred.



In this diagram, you see that there is an interdependency between these domains. In particular, there are processes inside these domains that temporarily pass control (perhaps asynchronously) to the provider domain, which in turn can pass control, etc. When I thought about this harder, I came to the conclusion that there were two views into this intersection: the Interface and Interaction.

The interaction defines the reason for the intersection between the domain. Such as Request to purchase item or inquiry of books with "Amazon" in its title. An interface, however, is the actual connection between the domains. A "request to purchase item" can be connected to a consumer via multiple interface. There might be human or systemic. Said another way: the reason for the connection from the consumer process is the same regard of the method (interface) of connecting.



It was at this moment, that I realized that the domain boundary became critically important for sanity. So, that begged the question, "What is inside of the domain and what is outside?" and "What really is a domain?"

My conclusion is that a domain is an unit of authority that accomplishes work, orchestrates that work, provides and consumes services (collection of work) to other domains and manages exceptions, situations inside its sphere of control. Interactions specifically are what connect the processes (orchestrator) to other domains.

What does everyone keep considering WS-* as SOA?

I was reading Dana Gardner's blog on WOA will soon eclipse SOA as most impactful business transformation agent. I am confused why there is even a challenge.

Even at Gartner's Gartner Application Architecture, Development & Integration Summit Dec 2008 in Las Vegas had two men I highly respect Roy Schulte and Nick Gall (blog) had a point-counterpoint discussion about WS-* vs. REST. Although it was very fun to listen and participate, I believe that when statements about WOA replacing SOA doesn't seem to jive with me.

To me, SOA is more about take functionality and making it abstract, network-able and find-able. I believe that it's goal is to abstract functionality to the level that business people can understand and use it. That is why SOA is an incredible partner with Event Driven Architecture, BPM, BAM, Mash-ups, etc. When a business person (I would dare say the new "programmer") can understand the command of our new paradigm, productivity and effectiveness will go through the roof.

Perhaps the discussion focuses more around which is a better protocol for getting at the information. REST or WS-*. In that context, there is a lot of worthwhile debate. But I don't think an answer either way would make "WOA replace SOA" I think they are on different layers of abstraction.

I read a presentation of Nick Gall's (don't have the reference) and it said "We made software because hardware was too hard. Now software isn't soft enough. What is softer than software?" This is the problem I want to solve. I believe that the coordinated application of EDA (CEP), SOA, BPM, BAM in concert with a business awareness found in Enterprise Architecture will lead us down to the solution of Nick's challenge.

NOREX State & Local Government Forum Speech

It's been a while since I blogged. I have let circumstances of life erode away the goal of writing 20 minutes a day. Thats not good! :)

Anyways, I was asked by a group called NOREX to lead off a forum on SOA. In order to better understand this, let me give a bit of background on NOREX. In essence they are facilitators. They have meetings and teleforums where they talk about technology without all the vendors pressing their individual goals.

I was told that most people attending this session would be still in the introduction or early implementation. This is my lead in for the SOA Teleforum. Hope it can benefit someone.

Jeff

What is SOA?
NOREX State & Local Government Forum
SOA Teleconference

As you know, SOA has been having a lot of hype lately and probably why we are discussion it today. SOA was coined by Gartner Analysts Roy Schulte and Yefim Natis and has been like gangbusters ever since.

But what is it? What is about this new “architectural style” that gets people excited and gets Gartner to say that by 2010, this style will be used in more than 80% of new mission-critical applications?

If we extend the Acronym, we get Service-Oriented Architecture. I’m sure you already realize that Service is the key word, but how does that apply. If you look at any other field, a service is something that is done for the benefit of another. You could look at this as an electrical or water service, or restaurant waiters or tax accountants. This definition is exactly what is meant by the service in service-oriented architecture. A piece of software that does something for another. However, all of the other examples have another thing in common: multiple consumers of their service or put another way, they were not created for the expressed purpose of one consumer. If we look at tax accountants, he or she follows the same set of rules, procedures, etc for every consumer that uses the service although each may take a different path inside the process based on individual circumstances. The tax accountant is consumer-agnostic. The service is not built for the needs of 1 consumer, but all or a large set of consumers. The Service as part of SOA is the same way. As an example, if we build a web-service that is designed to accept input from a single source, then it is not a service, but rather an API or Application Program Interface only.

So, let us take a look at the key principles an SOA must meet:
• Modular
• Loosely coupled
• Distributed or networked
• Contractual
• Consumer-agnostic
• Abstraction and understandable

Modular and loosely coupled are things that Computer Science professors kept drilling into our heads. The saying that I had to learn is “Maximize Cohesion while Minimize Coupling.” SOA enables both of these concepts. As an industry, we have been trying to get this going for a long time. In particular, modular programming, APIs, DLL, CORBA have all been trying to accomplish this goal. We are continuing the work.

Distributed or networked is very critical. A lot of the programs that have created over the years put the intelligence of the program deep inside. To understand this, think of a billing system or point of sales system. The functional piece of code that calculates sales tax is used over and over, but generally is buried deep inside of the code or accessible only via DLL or library. Distributed or Networked principle says that it has to be accessible via the network or more precisely that it is not loaded as part of the code base of the consumer as would be in a DLL or library.

Contractual principle is a very interesting one. Any real-world service has either an implied or explicit contract. When you drop your clothes off at the dry-cleaners, you determine a price via a chart; you choose a pickup based on rules; you choose starch and other options. The same thing applies with SOA. You have a contract that indicates what will be done, the functional and non-functional requirements, you indicate the semantic definitions.

Consumer-agnostic we discussed earlier, but in essence, you build a service for a large set of consumers not one consumer. That isn’t to say that you might not build something as version 1 to fit a single consumer and then increase its consumer-agnostic ability as we get into version 2,3 etc. This amount of this attribute is directly proportional to the ability to increase reuse.

I saved Abstract and understandable to last because it is the most important. In fact, I called SOA: Service-oriented Abstraction. Abstraction is the ability to make a complex item knowable. It allows us to conceptualize. It is this conceptualization that allows designers to use it. A computer example is the idea of a disk folder. There is a lot of complexity to being able to read and write to a disk drive. Years ago, device drivers were created that generalized this and made a common interface and then visual folders were created. So now we have only had to drop a file inside of a folder and the machine takes care of getting it written to the actual physical hard drive. Services allow the ability to abstract. The question is at what level of abstraction you build your services. We could build a service pretty low level and just make some functionality like save-to-disk or we could make our services higher level and make it something that the business user will be able to understand such as calculate sales tax or create order. By building higher level services, then we increase the opportunity to allow business users to be more actively involved and even to be building their own code.

As you can see, all of these principles build upon each other. It is the building that makes things

This presentation would be wrong if I didn’t at least say how we can use these services. Once we have built services, we need to figure out how to apply them. Obviously, we can use them inside of traditional application code. We can build applications that leverage services from other agencies or external entities. We could also use business process management (BPM) to define the business processes of our agency. By building higher level of abstraction services, business users can create or change their process to innovate. IT needs to be involved from assurance, testing, etc. perspective, but this will allow a better integration between the business and information technology.

Still other perspectives are web 2.0 concepts like mashups where people take functionality and data from different places and combine it in a new way. Perhaps by solving a problem that is high priority for them, but low priority for the enterprise.

You will notice that I never equated SOA with web services or REST or CORBA. I believe the concepts are not equivalent. The latter are transports or how I get it from here to there. But by using the principles I laid out, and then we will be pulling the functionality and data that are trapped in the bowels of our applications and bring it to the surface where we can build off of it and inter-collaborate better than ever before.

Contract-First Development of Services

Contract-First development of services will help companies move towards a service oriented architecture (SOA). When you can abstract code into services, it makes it easier to use those services as pieces and plug them into where ever they might be needed. By first building the service's contract, it provides the structure to define and make those pieces work together.

However, what IS a service? Some might say "it's a web service" or "It is creating modular, reusable code." It is true that the web service standards are important and re-usability is an important attribute. However, by taking the view that a service in an SOA is more similar to a lawn service than a web service helps debunk the notion that services are simply code that can be easily called from anywhere (Modularity). Instead, it allows us to consider that a service as a stand alone piece of functionality.

When you hire a traditional service such as a lawn service, the first thing that occurs is an agreement as to what will be done. The elements of this agreement include the name of what will be done, what the customer has to provide and what the end result will be. The agreement will also include performance and response time criteria. In the end, as a consumer, I don't generally care if my lawn care provider cuts my grass with a 20k dollar mower or a push power or a pair of scissors. The implementation is up to the provider, SO LONG as it is implemented meeting the agreement. We call this agreement a contract.

Services found in an SOA also need this type of agreement. A service contract includes both functional and non-functional requirements. The functional requirements include what the service does, the service interface and the means of invocation. the non-functional requirements include service level agreement, security and quality of service, transactional constraints, process and semantic definitions.

You might ask, "Why do I need this contract? What does it provide for me? Can't I just use the interface?" The reason you need it is to give the service it's own scope of responsibility. A basic tenet of SOA is that a service is an autonomous unit of functionality. Also, it provides a definitive scope of what that service need to provide. A contract increases visibility and provides a neat package of functionality that can be used as a component in our application infrastructure.

Another advantage in knowing the functional requirements is that before the code is even created, test cases can be developed to determine if the services performs the required functionality within the constraints of the contract. Some call this test-driven development.

Stability of the system is critical when you create a SOA. This stability is improved by the versioning of ontracts. Once a contract is established, it can not be changed and it must be supported as long as consumers exist for it. However, new versions can be cloned and modified. By having both versions of the contract supported, new functionality can be built and made available to early adopters while the old contract is still supporting other applications. This allows a company to be agile in reacting to business changes while not breaking existing applications.

Lastly, the contract should state ownership and responsibility matrix. This is in the form of who is involved as well as who is responsible for the evolution of the service. The RACI is a great starting place to show these stakeholders. RACI stands for Responsible, Accountable, Consulted and Informed. I have always had an issue between Responsible and Accountable. They seem very similar. So lets do the easy ones first.

Consulted represents the set of people who have input in to the contract. They are typically the main consumers or business subject matter experts. The important part is that this is a two way communication. Informed are the set of people who need to know about changes or the development of the contract, but do not have a voice in its creation. This is a one-way communcation.

Now for the difference between Responsible and Accountable. Responsible to me is the person who owns the problem or, hopefully, owns the vision. Accountable seems to be the person who has to approve it. One variation that I like more for the 'A' is assists. Who assists the Reponsible (Visionary/Owner) in making the vision a reality.

Assemble-to-order: How using it can improve agility and operational alignment

In manufacturing, there are several categories of production. A manufacture can be a "build-to-order", "engineer-to-order" or "assemble-to-order" shop. Engineer and build-to-order means they make their products from scratch or raw materials. This is the most flexible, but requires a longer lead time to make the final product and therefore is less agile. Assemble-to-order means that themanufacture take parts called assemblies, put them together in different interesting ways to create the mered of potential final products and thereby increasing agility. An example of this would be Dell Computer. When they take your order, they have lists of options for each area or functionality. These areas would include disk drive, motherboard, ram, operating system, monitor, etc. This ability allows Dell's customer to mix and match their system based on their needs.

Service Oriented Architecture gives IT shops the same advantage to their customers. By design, SOA helps engender the "assemble-to-order" mentality that is needed in order to become agile. In these terms, an SOA service is functionality. It abstracts or hides the complexity of "how to do something." So, this service is the equivalent to the assembly part in the manufacturing process.

IT shops have to decide on how to get these assemblies or functionality and what are the assemblies they need. The ability to make these decisions separates the successful implementers of SOA from the frustrated. Answering the how question better allows more agility. Answering the what question better allows more organizational alignment. Let's focus in on both.

The how decision affects agility because of resource allocation issues. With unlimited budget, talent and time, we would build everything from scratch because it specifically addresses our issues. Since this isn't the condition we live in, we have to decide what we build based on things that will make a difference for us in the marketplace and things that we have thecompetency to build. We need to allow others (3rd Parties) to build things that don't make difference for us or that we don't have the competency to build. A rule of thumb is from Anne Lapkin of Gartner , "Build for competitive advantage; buy or reuse for competitive parity." The beauty of this principle is that not only does it limit what you build, but also indicates who you should hire.

As we increase our understanding of the functionality needed to enable business processes, the more IT can serve the organization as a whole. Business are in essence a collection of processes, ideas and effort. Processes are broken into steps. Each of these steps is a function. An action to be taken. To help define the set of functional steps, an enterprise should develop a vocabulary. (There will be further blogs about vocabulary)

In the beginning, it may seem to have a lot of overlap and confusion as to what is a service. How granular to make the service. The rule is to make it large enough so that a typical person who deals with that process can understand what is happening but no more than that. This is what I am referring to as the Middle Ground between the Business and Information Technology. This is the chasm where chaos rules in today's business and IT integration. Semantics, language, comprehension of processes from differentvantage points all lead to this ineffective integration.

It is this chasm which causes a lot of the confusion. But it is also the place of the richest gains in productivity can be gained. I am not saying that to tackle all of the middle ground issue at once, but once familiar with the tools and how to use them is achieved, find an important but not mission critical process that lives in this middle ground to work on first. Use enterprise architecture vocabulary to name and describe the steps for the process. Then build, buy or reuse the services named according to the Build-Buy principle.

An important note to remember is that there may be more granular services that your higher level services use. The services in the middle ground are the ones that need to be understandable to the typical business person using that process.

Enterprise Nervous System

The Enterprise Nervous System (ENS) is an allegory to the human body. In the human body, there are two methods of reactions to various stimuli and situations: Conscience and sub-conscience. In this blog, I will attempt to bring together these thoughts and hopefully help make sense as to how events are the signals of the Enterprise Nervous System.

In conscience reactions, the brain determines a course of action based on its interpretation of the stimuli & situations and causes actions to occur directly. In sub-conscience reactions, the actions happen as part of conditioning or reflexes.

I would argue that systems that require a lot of human decisions have a higher conscience quotient which allows for thought before action, but reduces the responsiveness. Systems that are conditioned to respond in a certain way are more like sub-conscience reactions. They have the ability to quickly react to a situation without "thinking" about it.

As we have learned from Six Sigma techniques and process improvement efforts, there is value in conditioning responses. In particular, processes that are rote or have a preplanned responses should be developed using techniques that maximize this responsiveness. These would include using triggers for processes or business rules engines to make decisions. As they are reacting, they will execute services or "the muscle" of the Enterprise IT body.

However, not all responses can be or should be developed as automatic. In particular, innovations either by the company itself or its competitors can not necessarily be anticipated. These situations require the brain power of the Enterprise to work through how do/should we exploit or defend against the opportunity or threat that is reveled. This may require developing new muscles (services) or changing how the automatic reactions are occurring.

The goal should not be to automate everything for automation sake. The goal should be reduce the load placed on the conscience or thinking parts of the organization by creating conditioned responses to familiar or routine stimuli to free it to concentrate on non-routine and extraordinary situations.

Events are critical to this endeavor. Events carry the context and particulars of what has happened. When the event is something that we can in the sub-conscience sense, it is handled. If not, it should be escalated to the attention of the brain through a variety of tools. Business Activity Monitoring (BAM) and Complex Event Processing (CEP) are two of these mechanisms.

CEP allows events to be combined to infer other more complicated events. BAM allows trends and metrics to help determine when a situation changes. It is this change in the "State of the World" or significant event which typically need the attention of the Enterprise Conscience mind. It is important to note that these engines job is not to solve the problem, but rather to find the needles in the haystack that need attention.