Thoughts on integration

First let me say thanks to everyone who attended the annual C3 User Group Conference this week. It was great to see you all and share new ideas. The energy and excitement was contagious as always. I look forward to this event every year and this year didn’t disappoint. You are the reason.

At the conference, a major conversation theme was how best to address the integration challenge. This topic was so popular we even altered the conference agenda to make room for an entire session that allowed us to explore the challenges and approaches related to addressing the need to integrate the different solutions in a meaningful way.

At this session I was intrigued by the areas of concern and the aspects of the effort that posed challenges. In listening to the dialog, it occurred to me that what was missing was a way to think about how to approach a broad integration initiative that provides structure and relieves the pressure of having to think about everything all at once.

With the work we’re doing in the development of the Click® Product Suite, we must address these same integration needs in a way that meets the following criteria:

1. Support multiple deployment models – Each product in the suite must be installable into it’s own store or be combined with any other products in a single store. Decisions on which model to favor or how the products should be combined should in the hands of those installing the product and not dictated by the design of the product.
2. Independent with the ability to interconnect - Products should be able to stand alone and not require any other product to function.
3. Support extensible integration – We must recognize that the potential integration points extend beyond the set of Click products and a product should support the ability to integrate with any application through supported extensions. In addition, the information shared by a product for other products to consume should be extensible.

Integration is such a broad topic that the first step is to define what it is and what it isn’t. At it’s most basic level, integration involves the sharing of data between solutions. Data sharing though isn’t enough. Here’s how I think of the elements of integration:

• Data Integration – The sharing of key data between solutions.
• Process Orchestration – Events in one solution drive processing in others.
• Navigation Integration – User must have a way to easily navigate across the different solutions.
• User Interface Consistency – Whether there is one solution or many, the user’s experience must be consistent.
• Communication and Cooperation between product stakeholders – This element isn’t technical but is the most critical. If the parties involved can’t agree on what and how to share, then no technical decisions can make integration happen.

At the heart of any integration discussion is the concept of the sharing of data and product events. In the general sense, these are two types of messages representing the information that is exchanged between products. In a messaging model there is both a sender and a receiver but we have to be careful to avoid thinking about this in a literal sense. It’s easy to fall into the trap of imagining an array of point-to-point messages between specific applications. Thinking about it in this way makes it seem more complicated than it should be because it leads to a model where the sender of the message (be it data or an event) must  have knowledge of the recipient of that message.

This results in a product having knowledge of another in a way which created dependencies that makes it difficult to still retain separation between products. When it comes to building out a suite of products that can both stand on their own and integrate with other products if they exist in the installed environment we must be able to retain product autonomy. Anything else and we quickly degrade into a need to build the suite as one truly massive and unwieldy product. This is something we must avoid.

Publish and Subscribe instead of directed messaging
To avoid degrading into such a complicated web, a better approach is to build with the notion that the sender of the message does so without any idea of who is out there listening. Think of it more as a broadcast where the information of the products that tap into the broadcast is external to the message sender.

The messages a product broadcasts define its messaging API. This API is formally documented in the product’s definition and any other product can independently decide the set of messages to which it wants to pay attention. Thinking about integration in this way eliminates the need for the product publishing the message to have any knowledge of the destination of it’s message. It doesn’t care whether no one or everyone is out there listening and this allows us to focus on the actions and needs of each product without taking on the entire world of applications at the same time.

This approach to messaging will allow us to elegantly handle the first two types of integration: Data Integration and Process Orchestration. Click® Portal 6.0 will include the first phase of underlying messaging support for us to begin using in the construction of the next generation of Click® products. Early work in this space is very exciting and, though we still have a lot of work ahead of us, getting the integration platform in place is a significant first step.

We’re also sinking our teeth into the next two integration topics: Navigation and User Interface but I’ll save a discussion on those two for another time. As always, comments are welcome. I truly appreciate your feedback. I’ll have a lot more to say on this as we progress so stay tuned…

Cheers!