Different types of data integration
Integration continues to be a difficult subject and this is quite often because there is so much variety in integration needs and available solutions. Therefore it is important to distinguish what kind of data integration we are trying to solve. Making an inventory of all the integration requirements and a grouping per type
- One record (write / update / delete) - usually for data entry of transactions or reference data
- Retrieve one record (read - look-up) - usually for master reference data
- Retrieve one record (read - transfer) - usually for a transaction, but also to be used for master data if it needs to be replicated
- Combining data from different sources (read / combine / potentially transform / present) - usually for reporting
- A bulk data transfer (retrieve / transfer - including a potential transformation) - for e.g. data warehousing purposes, i.e. to support a better performance in a different de-coupled data model
- A bulk data transfer (retrieve / transfer - including a potential transformation) - for analytical and modelling purposes, where the data in the target system will be subject to change
Enterprise information architecture needs to deal with all these types of integration and usually it will deal with these different types in different ways. By understanding the different types and grouping the integration requirements per type it is possible to rationalise the technology supporting the integration.
Through this it is also possible to see that certain technologies only meet the requirements of a few of the types. For example SOA type architectures are usually good for single record situations (type 1-4).
Labels: integration
More on integration ...
As I wrote before, one of the main aims of Data Architecture is to support the needs for integration in the organisation. There are different ways to achieve this and therefore Architects are usually in favour of Big Projects. Developments like a Common Database (with a common data model) or a Data Services layer (with a canonical model in the service layer) are the most common projects. These projects are never popular and hardly ever a success.
At the end of the day most organisation live with a lot of point-to-point connections and because they are more practical, cheaper & faster to build they may be a better solution (even though we always way that they are less maintainable). So the challenge of the architect is to balance the cheap, fast ad hoc link with a nicer layered architecture. Usually the cheap thing wins and for a good reason ...
One way of achieving a solution for this is balancing act is to define a standard for the common framework and let projects develop their 'point-to-point' as part of this standard. Through publishing the specific link as meta data (with an XML description) we will see the services layer grow!
Labels: architecture, integration, middleware, SOA, XML
Integration is the Key
Things like data management and architecture are quite often still elusive subjects for senior managers. We produce pretty pictures, we make comments that are hard to disagree with, but when people ask us the 'so what' question we usually have a complex story about roles and responsibilities, compliance, implementation of difficult 4-dimensional data models and other intangible things. What we usually miss is the elevator speech that sells our story.
I have been reading through previous posts and I see a lot of things the fall in the first category I mentioned - the difficult & intangible overhead stuff. Therefore it is time now to focus on the one-liners ... I even think we can reduce it to a single word: The word
Integration. Probably the main reason why we do data management is because we want to integrate something:
- Integrate business process flows
- Integrate information from various processes into a single management view
- Integrate information over time - so we can compare today with yesterday
- Integrate the inside with the outside
- Integrate data with documents, maps, ...
OK - there's a few other things - risk management, compliance, etc. (pretty important stuff), but they key thing that people understand is the word integration. So probably in the future I am going to drop the word data management in discussions. When people ask why when I propose a certain measure, than I will say: Because you need to integrate ...
Labels: culture change, integration
