I have often been asked to describe what are technical parts of an Integration Platform – what they stand for and what why they exist. I have also noted, that most of the major players lack surprisingly simple features while having advanced functionality and support for hype-status technologies. But before deep-diving into those, in this post I’ll describe briefly must-have features of today’s Integration Platforms in chapter named surprisingly: “Must-have features“. If you are already familiar with most common concepts under the hood of an integration platform and are interested in more rare but yet crucial features, jump right through to chapter “Crucial, but yet missing features”.
Message Bus ensures secure and reliable asynchronous communication across enterprise network. Message Bus has ability to route messages between sources and receivers. It queues messages if receiver is temporarily unavailable. and There are several models how managed message processing can be done – publish/subscribe (pub/sub) model seems to gained most of the ground in today’s integration platforms. Quite often term Enterprise Service Bus is considered to include major parts of Integration Platform – don’t mix mere Message Bus to ESB.
Adapters / Connectors
Applications – like Line-of-Business systems including ERPs, CRMs etc are connected to Message Bus via Adapters or Connectors. Adapters transforms the proprietary interface to compatible internal form for Message Bus. For example, major integration platforms include SAP and Siebel adapters out-of-the-box. Cloud integration platforms include adapters to cloud-native business applications like Salesforce. Sometimes connectors are implementation of certain standard technology and/or protocol and offer connectivity to other systems implementing same standard or protocol, such as SFTP, HTTP , EDI etc.
Transformation engine runs protocol and format transformations to messages in bus or raw files in file system. Most common transformation markup is XSLT – which is usually hidden behind visual mapping tools. For example, when an application has sent a message via Adapter to Message Bus (e.g. published a message), message is probably in XML format. Before message is usable by receiving application (subscriber) it usually must be transformed to some other XML or flat file format.
- XML to XML
- delimited to XML (and vice versa)
- fixed field length to XML (and vice versa)
XML to XML transformation is the most common need today, but it won’t hurt to have flat file (delimited, fixed length) transformation ability for old legacy systems.
Metadata repository has process related information that should not be hard-coded in the process. For example, business party information change between separate instances of purchase order process. Metadata might as well be more technical data snippet that is specific for a party or group of parties. For example, one “Push basic data to Point-of-Sales” process metadata might contain URLs or FTP addresses of certain sales point. Group of sites might have different POS systems which need dedicated transformation etc. A good integration platform offers ways to manage individual points and group metadata intuitively together.
Business Process Orchestration
After integration platform include reliably transferring of messages between enterprise applications, can business processes be orchestrated. Process orchestration can be thought as series of sequential actions with logical “and” and “or” paths. These decision may include automated business decisions. For example, when “Order” message is received during “Ordering” UBL 2.1 process and buyer party information extracted, process can ask credit rating, check inventory and so on calling for example generic services of service layer (SOA). Process path may branch to other path, depending on answers to these questions. “Order” may be rejected if credit rating is not valid or inventory says that products ordered are discontinued. Sometimes human decision is needed to choose the path of the process. In these cases integration platform waits for the application, which UI is used to receive the user decision.
An instance of a process that orchestration technically represents, may last running for months. These kind of processes are usually called as long-running processes. As common challenge in System Integration is pace and quantum of changes, it is likely that sooner or later an orchestration in running state is changed as well. Well designed integration platform is ready for this kind of situation.
Every Integration Platform should contain technical monitor. Monitors typically have dashboard like view to all messaging and orchestrations that have run during last week, month or any time-windows that integration platform is set up to keep history data. They also should warn on on-going exceptional situations like message throttling on peak message volume times. For example, as a result to a successful marketing campaign, amount of orders may suddenly rise to a level that integration platform can’t process normally – it starts to intentionally slow down receiving process so that it ensures that every message is persisted properly and server is not over-stressed otherwise.
Business Portal / Data collecting
In my opinion, it is quite questionable that Business Portal – like BAM in BizTalk – should be thought as an integral part of integration platform at all. In most of the cases it can’t compete with separate products dedicated for business intelligence. For example, Microsoft’s own BI tools outperform BAM. In the other hand, some users use BAM just to get better technical view to an orchestration on BizTalk.
Some new integration platforms include automatic or semiautomatic big data collecting to cloud from data flows that run through integration processes. Built-in big data collector opens many possibilities to create superior BI solutions. What would be better place to store data in decided state than integration platform through which business data naturally flows?
Rare, but yet missing features
Lets start with an example: employees love to get their salaries on time. “Salary Payment Process” is a good example of a process that requires scheduling like “1st banking day of month“. There are many others processes with similar needs. For a long time Integration Platforms are thought to be just messaging platforms – or they are just focusing on it. What about good old-fashioned scheduled batch jobs? They are inextricably part of enterprise processes. Still – we need to have some separate special systems like Control-M or Automate or anything other than Integration Platform to schedule, monitor and run batch jobs. Why? A heavy scheduled batch job that launches a script that exports product catalog in XML format and sends it to multiple business parties via SFTP. Isn’t that a process that should be taken care as a one entity?
In Microsoft world we need separate add-ons, like Forsythe’s and Pereira’s Scheduled Task Adapter or FRENDS Iron to manage scheduling. MS has also just published Azure Scheduler preview as independent service. What good integration platform has is a centralized view to all processes, no matter whether they were launched via external event like incoming message or schedule. Some platforms include features like “run once per day” and “run once per hour“. I wouldn’t account these as a scheduling. Real scheduling mean ability to include external calendars and more complex schedules like “Run every bank day of a month” or “Run 25th day if banking day, else run next banking from 25th day“.
There should exist only one system to manage process scheduling – otherwise we’re on our path to spaghetti scheduling where processes are triggered by integration platform, Cron, AT commands, SQL Server’s scheduler etc. Synchronizing separate processes will become fuzzy task to manage and monitor.
I would declare batch-like workflows and jobs – scheduled or event based – should be integral part of integration platform. For example, imaginary “Export product catalog” process, which
- first runs a scheduled remote export script on a legacy product management system server, which locates it to FTP server
- pull huge export data from FTP server to local working directory
- enrich data from several sources
- send to background DB of Web shop
In this over-simplified example steps 1 and 2 are usually managed by job scheduling system to avoid persistence of a large file in message bus. Steps 3 and 4 are run on event based integration platform. Still, all steps are sequential steps of this job are one technical process and for the sake of audit-trail, unbroken sequential steps and monitoring they should be controlled as one. Same need goes for managing ETL operations, workflows in workspaces etc. These operations are often part of an other process and companies are using one system for scheduling, another for batch jobs and third for messaging – and this just breaks the continuity of sequential steps and add complexity for monitoring.
I would go for a single integration platform, which not only offers messaging and features described under “Must have features” chapter, but also is capable of batch processing, controlling external ETL tools (like SSIS in MS world), large file transfers and scheduling as a inseparable part of processes. This way company ends up with single system with centralized view and management of every technical process and good means to build real live business monitoring.