More on integration ...
As I wrote before, one of the main aims of Data Architecture is to support the needs for integration in the organisation. There are different ways to achieve this and therefore Architects are usually in favour of Big Projects. Developments like a Common Database (with a common data model) or a Data Services layer (with a canonical model in the service layer) are the most common projects. These projects are never popular and hardly ever a success.
At the end of the day most organisation live with a lot of point-to-point connections and because they are more practical, cheaper & faster to build they may be a better solution (even though we always way that they are less maintainable). So the challenge of the architect is to balance the cheap, fast ad hoc link with a nicer layered architecture. Usually the cheap thing wins and for a good reason ...
One way of achieving a solution for this is balancing act is to define a standard for the common framework and let projects develop their 'point-to-point' as part of this standard. Through publishing the specific link as meta data (with an XML description) we will see the services layer grow!
Labels: architecture, integration, middleware, SOA, XML
Architecture and Information Services
I strongly believe that most organisations today are mostly dealing in information and therefore - building a layer of 'information services' is something that should be considered when trying to build a sustainable and agile architecture.
What is the story of information services? Information services are a way of decoupling data from functionality. In the past this was already done by creating databases - but usually the databases were application specific. In the mind of most programmers there cannot be data without an application.
When applications started to grow and started overlapping, then the concept of shared data emerged - this happened at the end of the 'eighties' and suddenly people started to think about designing major data models. One of the most glorious attempts was the 'Epicentre' datamodel - designed by the Petrotechnical Open Software Corporation, an object relational database for the oil industry with more than 2000 tables. It was very clever, very flexible and very hard to implement. And therefore it was never a success.
Obviously these heroic attempts were doomed and with the growth of off-the-shelf applications we saw the rise of ETL tools (to get a grip on all the interfaces) And quickly after the rise of ETL, with the rise of the Internet, we saw a complete market growing of middleware solutions.
As information managers found out - it is also possible to create spagghetti with middleware solutions as well and now eight years after the millenium we are back at where we started - we are crying out to get more control on the information.
And this is where the information services story comes back again. If you put information at the center and 'wrap this up' as a service, using a canonical data model (a bit like the Epicentre data model, but then more pragmatic ...), then it is possible to reduce the complexity of your architecture. The data can be in various places (so don't think about large corporate data stores) and the middleware just 'knows' where to get it. The applications just 'shop' for data from the middle layer and the 'bus' is doing the work. Well this is where the complexity is - building the 'bus' is the hard work - since it requires
- rationalisation of data sources (agree where is the master)
- common definitions (or at least agree how to transform)
- routing (agree how the data flows)
And this is where we in IT are not good at - it is great to define a lot of new functionality, but to crawl through the sewers of data management is very hard. But if you want to succeed in getting control on your information centric entreprise, then think 'information services'!
Labels: architecture, middleware, SOA
Architecture in different dimensions
One of the mistakes I see on a regular basis is that designers try to fit too many different dimensions in one architectural design. Therefore in this post I would like to plead for a few standard views that describe the architecture in different dimensions.
In my view I would plead for a high level data flow as the basis of any data architecture. In this flow you can see the main data stores and interfaces. The data stores should fit in the data architecture patterns described a few weeks ago. The data flow can be divided in the main layers for e.g. corporate / master data, project data / data marts, archives, etc.
On top of this view you can have multiple solution architecture views for the technology stack related to a solution. This should include the main data stores at the bottom (in line with the main high level data flow), middleware in the middle and applications + viewers at the top.
Providing these different views will help the discussions on data flows (e.g. where is the master?) and the actual technical implementation of a particular solution.
Labels: architecture, master reference data, meta data, middleware
Middleware
Since the day we started moving from two-tier to multi-tier IT architectures we have seen a growing complexity around how we integrate applications, data, etc. Through this we see a growing number of 'moving parts' emerging in every implementation stack. It increases flexibility, once up and running it will make deployment of changes a lot easier, it even helps with scalability, cost control and ..., but it becomes pretty complex.
The so-called
middleware market (covering hubs, connectors, real-time services, etc.) has been growing a lot and quite often I see that we need integrators to integrate the integrators (just to add to the complexity) and that's why it is important to really work on you architecture.
The rules are actually quite simple (and very old fashioned)
- Keep it simple (and this is rule nr 2 and 3 as well)
- Standardise - try to have one technology for the messaging and data transformation
- Do portfolio management (limit versions, vendors, etc.)
- And continue with keeping the overview of how data flows from one place to the other. This will help with challenging if another point to point interface is needed (why not channel it via a hub?)
- Always challenge the next middleware technology, because quite often there are simpler solutions possible (e.g. if you can solve it with data integration in the database, why have a messaging hub?)
- And if you have to choose a technology, than choose an open standard using XML, because that will keep your options open. Especially for integration with the outside world there is a lot of merit of using XML, since it is self-describing.
If you don't do these things you will see spiralling costs, the need for consultants with exotic skills and at the end of the day an unmanageable architecture ...
Labels: architecture, middleware, XML
Information Architecture
In my previous post I mentioned the word
'architecture' and this is a bit of an elusive concept in Information Management. What does it mean?
One of the most common references in this field is the
Zachman framework - a very complete framework that covers data, applications, infrastructure, people, processes and even motivation in all its aspects (from high level to detailed implementation). I think it is a great concept for understanding all aspects of IM&T and it covers a lot I have been writing about.
If we just focus on the data architecture, than the framework is a bit large. It is very easy to lose track in all the things that need to be analysed and documented (before you know it you spend more time on the framework than on improving data management - paralysis by analysis), so therefore I would like to focus on the four main elements of information architecture that are important to me:
-
Master reference data: large organisations need to establish as much as possible the master sources for their key objects. So one place to manage people information, product information, customer information, etc. From these master sources this information can be shared. Master data requires common definitions and clear ownership of information.
-
Middleware and data integration: large organisations need to define clearly how information flows from system to system. This is to avoid spaghetti integration. Different integration concepts are possible (via a central data store, via a middleware layer, etc.)
- A supporting
Data Management organisation: An architecture needs to be owned & maintained. Just like a garden needs a gardener. The Data managers take care of establishing the blueprint, the standards, ensure quality is measured & improved and obviously they take care of the day to day operation of the data stores and interfaces.
- A
high level story: To establish all these things requires sustained data management investements and this can only be achieved with sufficient sr management commitment. So the data architect also needs to have a high level story on what architecture can achieve and what improvements (successes) have been made.
Labels: architecture, master reference data, middleware
