Service-Oriented Architecture

When designing your SOA and services, keeping the service consumer in mind will make the job easier. Consumers must conform to the interfaces of each service they use and invoke them with the right data in the right format. The more similarity there is among services, the less coding and translation your consumers will have to do. Using the techniques of transformation, semantic data modeling, and a conceptual data model can make your job much easier - both during initial design and testing and when making changes later.

The interface specifications, protocols, and data formats used in SOA and Web Services are designed to create services with "loose coupling" between the service consumer and service provider. The service provides a consumer with an abstraction representing some business function so that the consumer doesn't have to be concerned with the details of how the function is implemented or how and where its data is stored. The only thing that the consumer has to know is how to call the service using standards like WSDL, SOAP, and HTTP.

Once you've implemented some services you can then compose higher-level business functions and so-called "composite applications" by invoking multiple services and orchestrating interactions among them. If you build all your services from scratch for a project you can design interfaces and data models that interact smoothly and consistently. Opportunities for such green-field designs are rare and today's new system is tomorrow's legacy system anyway. More than likely, you'll be using existing as well as new services and applications that were bought or built long ago and have or will have service interfaces added to them.

The Fly in the Ointment
An application's service interface may be new, but it probably won't be clean or consistent with your other services since an application's functionality and existing interfaces usually dictate what can be provided as a service, as well as what data it can accept and provide. Interacting with multiple services can become complicated - even though they use standard protocols and formats for interface definitions and invocation, these protocols and formats only deal with the syntax of the data and not with their vocabulary or meaning. Each application or service is likely to use different names for the same things and the same name for different things. Such disagreements and mismatches are commonplace and not limited to Service Oriented Architectures. Here are a few examples:

• Many of us have an address book on our computers, another on our cell phones, and software that synchronizes them. Unfortunately the definition of an entry in each is different, with variations in field names and the number of fields. One may have fields for home, work, cell, fax, and modem numbers and an e-mail address as well as the other home, work, mobile, pager, assistant, and main numbers and multiple e-mail addresses. The synchronization software knows how to map the different names but it can't map all the fields one-to-one. Still it's a good enough compromise because most people don't use all the fields.
• Many companies provide access to an external CRM system - like salesforce.com - for salespeople and use an internal ERP system for processing orders. Those two systems keep different information about customers so using the salesforce.com Web Service interface to get customer information and then correlating it with data from ERP system may not be straightforward. Like the address book, the data formats are different, but so is the intended use and content of the data. One keeps track of things like prospects and their interactions with the sales rep and the other keeps track of things like what someone has bought, how much they paid, and when.

The two systems both have a notion of "customer" but one may call the customer identifier "cust_num" and the other something like "csidno" and one may contain a number while the other contains both letters and numbers. One may have "orders" represented as an XML document with elements for "orderlines" and the other an "orders" XML document with the elements "orderheaders" and "orderdetail." Other data elements will present similar issues. Reconciling or mediating the differences in data models and service interfaces for applications like these is more difficult than reconciling differences in address books.

The consumer bears responsibility for providing the correct data elements to the service in accordance with its interface definition. In many situations, the service designer gets to decide what it wants to use. When you design service interfaces, you should try to make it as easy as possible for the consumers to use. That isn't always possible, especially if you use services provided by someone else like another department or an external organization like salesforce.com, UPS, or Google.

Transformation to the Rescue!
To mediate the differences among services' data formats you can introduce a data transformation or mediation layer into the architecture. This layer becomes responsible for mediating data formats between service invocations, performing conversions and translations for the consumer. The consumer then has the data in the right format when it wants to invoke a service using data supplied to it by another service.

You can construct the transformation operators using XQuery, XSLT, or even hand-coded Java functions to translate one data layout and vocabulary to another with relative ease. If you're using an Enterprise Service Bus (ESB) as a backbone for reliable communication and service management you can put your transformation services on the bus. The ESB can then automatically invoke the transformations as needed.

When you build transformations one by one as you need them, the result is a set of transformations for converting data received from service A into the right form for sending to service B, service A to service C, service B to service C, and so on. Each transformation does one specific job in isolation from the others. Now everything will run smoothly, right?

Transformation Isn't Enough
While transformation is necessary and helps to make service integration easier by itself it's not enough. Even when you use effective tools like DataDirect's Stylus Studio or Altova's XMLSpy to create them, having many such point-to-point transformations can make things complicated:

• It reintroduces the tight coupling SOA was supposed to eliminate in the first place. Instead of providing agility your SOA will become brittle and difficult to change.
• We've spent more than 50 years inventing new applications and their data models so there can be many complicated mismatches between applications. Every organization has a lot in their systems. Dealing with them can keep you busy for a long time.
• It's pretty likely that you'll end up with duplicate sections and duplicate operations in your transformations perhaps with unintentional variations, especially if different people make them.
• Maintaining hundreds or thousands of transformations is a big burden and keeping them all correct, consistent, and current will be next to impossible.
• Individual transformations encapsulate knowledge of the data semantics in many different places. That knowledge can't be found easily or reused effectively.

Transformations certainly help but they aren't enough.

Conceptual Data Models to the Rescue!
Instead of many individually designed point-to-point translations, if you define mappings from each service to a common conceptual data model then it will be possible to generate the transformations you need automatically and use them where needed. This can greatly reduce the amount of work you have to do because instead of defining transformations for pairs of services, you define one mapping for each service. At this point, the necessary point-to-point transformations are generated from the mappings from service to conceptual model.

Note that there's no need to actually transform to and from the conceptual model (that's why it's conceptual, not real). With the right tool for defining and extending the conceptual data model and service mappings and storing them, this approach solves a number of the problems we mentioned previously.

As you define and extend the conceptual model, you can use clear and understandable names for all the data elements no matter how cryptic they may be in the service interfaces. When the same data element is used by multiple services, the modeling tool can show them together enabling you to see where an element is being used even when the names are different.

Data Validation
The service interface standards we have now don't include much ability to express the rules or constraints that must be applied to the data presented to the service. For example:

A retail customer may not be allowed to use purchase orders, while commercial customers in good standing can;

There may be several related fields that must be validated together, such as street address, city, state, and ZIP code, and so all of them must be present;

A shipper can't be assigned to an order until a shipping address has been specified; and

Two addresses - current and future - are required in a move order for relocating broadband cable service. A service always has to validate its inputs but, in loosely coupled systems, it will often be necessary to do data validation of consumers as well - especially in service consumers that interact with users who want early feedback in data entry, for example, before an order is submitted.

The common model can be used to capture such constraints so they can be applied everywhere they're needed in a uniform manner. When the validation logic is generated from these rules you can be sure they're the same everywhere.