Concerning Integration, EAI, SOA ...
The vast majority of my work in IT has been concerned - in a broad sense - with the integration of systems. Few systems exist in isolation. This applies in particular to PLM systems, my area of experience. These never exist in isolation - PLM systems are central for any company that designs products - they most often as data hubs - the source of the data concerning products. The integration of systems requires hard work. Over the years I have seen and applied all sorts of technologies such as batch file transfers, screen scrapping, RPC, DCE, CORBA, integration hubs, EAI, SOA, ESB. But in spite of this, integration and the synchronization of data between systems, remains hard work.
I have spent quite a bit of the time of the last years fighting on global integration projects - mostly using SAP PI as a platform. An interesting experience with a lot learned. At some time I will get around to summarizes my experiences here. As always, and as in all other areas of software development there is no silver bullet to solve the integration challenges. But my experiences on this and other initiatives this year strengthen my belief in simplicity (configure in one place and in the right place) and in the importance of simple but adequate log files. In this respect SAP PI is a disaster. I just want a simple overview of all transactions going through - one line for each - with status and pertinant transaction data. Instead PI offers lists of XML received and processed, and for each you need to drill down to get to the data of interest: a sub-optimal solution.
For me perhaps the most interesting development (disregarding for the moment cloud computing, virtualization, Enterprise Web 2.0, Web Oriented Architecture, and all the other hype) is the steady progression and improvement of various open source initiatives - including those from eclipse SOA, SOPERA, Apache ServiceMix, JBOSS. Open source may greatly ease the issues of vendor lockin, may provide the basis for long term, adaptable, affordable solutions.
SOA: Save our architecture
Read the SOA Manifesto Many thanks to Nicolai M. Josuttis for the excellant book SOA in Practice - The Art of Distributed System Design and just as I was losing hope for SOA many thanks for explaining just how SOA should work.
In 2010 I attended a TechTalk in Berlin featurng Amazon's CTO Werner Vogels. This was an excellent talk and what impressed me most was his take on SOA. This point is repeated in the following interview in InformationWeek ... It's not just an architectural model, it's also organizational. Each service has a team associated with it that takes the reliability of that service and is responsible for the innovation of that service. So if you're the team that's responsible for that Listmania widget, then it's your task to innovate and make that one better. Around the time that I joined, we also did a deep dive on what these teams are actually spending their time on, and when we did an analysis on that, we found that a lot of those teams were spending their time on the same kind of things. In essence, they were all spending time on managing infrastructure, and that was a byproduct of the organization that we had chosen, which was very decentralized.
Also been going through Joe McKendrick's Top 10 SOA posts for 2009. Strongly recommend Joe's Blog. See also David Linthicum's Top 5 cloud computing predictions for 2010.
Reference: Joe McKendrick: SOA and the Cloud
Sadly looks like SOA is rapidly falling off the hype curve - just as realistic and promising approaches appear. Perhaps for me an important is the realization is that SOA in not a product - but a long term architectural strategy. In the works of the SOA Manifesto ... Service orientation is a paradigm that frames what you do. Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation ...
- Business value over technical strategy
- Strategic goals over project-specific benefits
- Intrinsic interoperability over custom integration
- Shared services over specific-purpose implementations
- Flexibility over optimization
- Evolutionary refinement over pursuit of initial perfection
The following are just some issues that spring to mind in the years of working on integrations. Most of these points really need further working on - more structure. Note: there may be contradictions in the points listed below (centralized vs. distributed, point-to-point vs. hub, standard adaptors vs. own developed, synchronous vs. asynchronous, standard vs. proprietary, configuration vs. customizing ...). These are mostly universally inherent to the nature of IT architecture. As always, a careful balance is required - avoid religious wars - do not polarize.
Reference: Prof. Alfred Katzenbach, Leiter Engineering IT, Daimler AG, ProductLife 2010
- Establish standards for: technologies, data formats, documentation, naming standards, monitoring frameworks, governance, etc ... Reuse as much as possible.
- Avoid architectural religions - always ask the question: if we buy, deploy, redesign, reimplement on the basis of this particular EAI/SOA product will this enable our enterprise, our business to sell one more of our own products ? Is the fixation on this particular technology the correct long term solution ?
- Accept that most integrations are point-to-point. Reuse of adaptors is in reality limited. This lies in the nature of what each system does. Each performs a different task in an enterprise. A CRM system has different integration points/requirements than a HR, PDM or EAM system. SOA does not necessarily mean EAI hub. Sometimes point-point is more efficient in every respect. Consider the implementation for each use-case individually.
- End-to-end connectivity is beneficial - system X needs to query system Y in order to determine what to do - decoupling sometimes leads to sub-optimal integrations
- Limits of graphical mapping editors: if the fancy graphical tool does not support what I actually need to do - then what point is it ?
- Introducing EAI/SOA brings a third party into the scenario - getting 3 parties to efficiently come to a working solution is a lot slower than with 2 parties
- Transparency of integrations - enable the business users to see what is going on - what has happened, where errors occur
- Hub + Spoke and analogy with the US aviation industry: back in 2000-2001 when I worked in the states and travelled by plane each week across country - the similarities between EAI and the US aviation industry became clear to me. The established United, American, Delta, etc all worked with a hub and spoke model (like EAI) - with United you had to fly over Denver or Chicago. The smaller upcoming dynamic, efficient, cheaper airlines flew direct routes point-to-point. What happens when there is snow storm in Chicago, or thunder in Denver ? Which did I prefer ?
- Sad realization after many years of fighting: perhaps the best interface is no interface - consider simplifying the application landscape. In my experience even the big software system suppliers whilst proclaiming they are "SOA based" or provide "standard connectors" to system XYZ, tend to treat integration as an unecessary evil - something that they only do because marketing want to see this option on the proce-list - but in reality these options are half-heartily implemented.
- Standard connectors/adaptors - how often have you heard from the sales or project lead from the supplier of software system XYZ that words "No problem, we have a standard adaptor to system ABC - we'll send our partner ...". Standard adaptors are exactly that and will - unless the partner has done a really good job - just that. They support only scenarios that they have though about - rarely these will fit your actual requirements. By the time you have written dirty work-arounds, user-exits, destroyed the clean architecture - then in the end you may have been much better off starting from scratch.
Futher topics to elaborate on ...
- Why is integration so difficult ? Political issues - issues of data/system ownership and control
- Are XML, SOAP, REST, Webservices the silver bullets for integration ? (there are no silver bullets !)
- Whatever happened to ODP: Open Distributed Programming ?
- For and against XSLT ... is this just another - complex and difficult to read - programming language ?
- Why are standard adaptors rarely universal ?
- Which hub ? Who is the master or mother of all hubs
- Scalability - hub or no hub / centralized bottleneck or distributed approach
- There can only be one master of data (you can't beat the laws of physics, or even temporarily turn them off)
- The challenges with security: authentication, authorization across system boundaries ...
- Metadata / Semantic integration ...
- Enterprise workflows - who needs yet another workflow system ?
- The problem with standards - quoting Andrew Tannenbaum: ...the nice thing about standards is that you have so many to choose from ...
- The reality of standard application APIs: incomplete, undocumented, over complex, single-threaded, unstable and buggy ...
- What are good services ?
- Granuality of services
- Universal services
- Service interfaces
- Adaptable interfaces
- Services for PLM
- Towards zero maintenance integrations
- End-to-end arguments: integration, transactions, error control, reporting, monitoring ...
- Transactions and consistency: good or bad / coupling transactions
- Synchronous vs. Asynchronous communications / integrations...
- Limits of throughput - parallel or sequential processing
- Issues with sequentialization
- Aim for idempotence - same message can be resent without causing double bookings
- Automatic retry - not all errors can be handled in same manner - in some error cases an automatic retry makes sense
- Where is business logic implemented ? In how many places is customer specific business logic implemented ? Perhaps implementation in one place makes more sense (in terms of solution maintainability) and perhaps implementation in the EAI tool is not always the best place.
- The forgotten usefulness of pragmatic scripting languages (Perl, TCL, ...)
- The undervalued importance of log files. There is an art of ensuring the correct content of log files. Just enough information.
Sadly most standard integrations provide either too much, too little or worse still too much data and no information. Examples of bad log files are:
- Those with thousands of lines of java stack dumps that end with: reson for failure: null.
- Those that do not echo exactly what the input data was.
- Those that you need to drill down multiple levels in order to find the pertinant input data (e.g. see SAP PI !)
- Dealing with errors in transaction postings: essential is the ability to repost data (good example here is the IDoc monitoring tools that have been in SAP R/3 since I have known it).
- Notification of errors in transaction postings: simple configuration of error notifications to business
Books / References ...
- SOA-Expertenwissen
- Dirk Krafzig, Karl Banke , Dirk Slama: Enterprise SOA: Service-Oriented Architecture Best Practices
- David Chappell: Enterprise Service Bus
- Nicolai M. Josuttis: SOA in Practice - The Art of Distributed System Design
- ...