More on integration ...
As I wrote before, one of the main aims of Data Architecture is to support the needs for integration in the organisation. There are different ways to achieve this and therefore Architects are usually in favour of Big Projects. Developments like a Common Database (with a common data model) or a Data Services layer (with a canonical model in the service layer) are the most common projects. These projects are never popular and hardly ever a success.
At the end of the day most organisation live with a lot of point-to-point connections and because they are more practical, cheaper & faster to build they may be a better solution (even though we always way that they are less maintainable). So the challenge of the architect is to balance the cheap, fast ad hoc link with a nicer layered architecture. Usually the cheap thing wins and for a good reason ...
One way of achieving a solution for this is balancing act is to define a standard for the common framework and let projects develop their 'point-to-point' as part of this standard. Through publishing the specific link as meta data (with an XML description) we will see the services layer grow!
Labels: architecture, integration, middleware, SOA, XML
Canonical Data Model
The main challenge continues to be the data model. In SOA speak: the canonical data model. Through some sort of difficult reason it is very hard for most IT people to deal with data models. Obviously they are not so sexy as the functionality in the application. Unfortunately it still requires quite a bit of skill to develop one ... and expecially an intermediate data model needs to fit a lot of requirements. It needs to...
- Support the superset of requirements of the related systems - and therefore it is important to model cardinalities, identifiers, attributes, etc. in the most flexible way. Cardinalities need to be flexible (by default many to many). Identifiers need to be unique (by default meaningless numbers). And with every attribute it needs to be clear that this is an attribute and not potentially an entity ...
- Support the reference data over time (4D), i.e. everything needs to be time stamped and no reference data gets deleted ...
If developers want to meet all these requirements, than they need to dust off some skills we dropped during the nineties.
Labels: meta data, SOA, XML
Enterprise Service Bus
The flavour of today in integration architecture is the Enterprise Service Bus (ESB), which is in a way nothing new. The idea of integration via an intermediate model has been around for some time and has led to many disappointments. But maybe this time there is a chance to make it work, since technology has moved on.
So what's new? First of all XML has matured to some sort of lingua franca; gradually all players in the market start to support the paradigms of SOAP, XML schema's, message based services, etc. And that means that momentum is building up. Second - the main difference is that everything is loosely coupled and therefore the responsibility for integration is now firmly put at where it belongs - with the applications. With other words - it is not the middle layer that needs to take care of everything, and this may be a factor determining its success.
Labels: architecture, SOA, XML
De-coupling data from its usage
One of the trends in integration architecture at the moment is the drive for a Service Oriented Architecture (SOA). For years we have been trying to rationalise applications and databases and time and time again we realised that with every migration we had to migrate a lot other stuff as well (like the increasing number of interfaces). Moving one application to something else would also lead to moving a lot of interfaces, reports, dashboards, you name it.
So now the idea is to hide all the complexity behind an abstraction layer. Why not publish all information via Web services, so everything below this becomes easy to replace with something else? 'All' we need to do is to describe the way want to see the information. The best way to do this is via defining a flexible data model (through an XML schema) and this is where the new trouble starts, because haven't we tried to do this before at the database level with enterprise data models? The good news is that this time we have more chance to succeed, since the middleware layer is much more flexible, since the data is not stored in the format of the schema, it is just described that way. And when needed you may even have more than one schema decribing the same information, without the need to store the data in a different way.
It looks promising, but we have a long way to go, since we only have started ...
Labels: architecture, SOA, web services, XML
Unlocking value
Data Management is a bit of a boring subject to many, But when done well it can lead to massive benefits that you cannot even imagine at the start. An example is the success of the Internet. At the end of the day it is actually a data management success. The
W3C has been very successful in setting and enforcing a few standards on how to address machines in a network (IP adresses), how to exchange information (the IP Protocol), how to name domains (DNS) - and replicate this across the network, and how to publish and link information within these domains (HTML). Everything else after that is now history. Nobody could have imagined the growth of the Internet in the last decade and half.
What it proofs is that we need to agree on certain basics and at the same time need to give the freedom to everything else. Within data management we see people spending enormous amounts of time in trying to come up with the perfect data model, or have endless discussions about how to define a unique identifier, but to me these discussions are usually a bit academic. What really counts is that a decision is taken; a standard has been agreed. It can be a standard which is not perfect (like the Qwerty keyboard); but if it works it will have benefits beyond what it was designed for.
Think about what would be possible with the wide adoption of XML to the same level as the adoption of HTML? Unfortunately agreement on formatting is much easier than agreement on content. Still I can see enormous steps being taken in the Web2.0 world where information from different sources is being mashed together from lots of different sources (e.g. Google News). Imagine what you would be able to do if information within a company could be mashed together automatically, because the information is easily recognised? So it is still worth the effort to spend some time on meta data, because it may lead to some unplanned by-effects beyond imagination!
Labels: meta data, standards, XML
The limits of Freedom: Manage your Data
The points made about supporting maximum freedom for the users are mostly true for Document Management, but when we talk data than we should be careful about allowing too much freedom. Data is by definition a more structured type of information than a document and structure means the requirement for control.
In a well functioning data management environment we have data sets that are used in more than one environment and therefore it is of paramount importance that this master reference data is of the
best possible quality and adheres to
clear and common definitions. Even better - it would be great to have a number of tags with every data item, telling what it is and telling you what has happened to it.
This is actually in line with some of the Web2.0 thinking. For document management the web2.0 looks like reducing the controls, but for data management this is not true - since this is about adding more context. And this is where the ideas of the semantic web come into play. This idea of the semantic web is about putting information into context so the meaning of information (the semantics) can be understood by machines (search engines, integration engines, etc.). Integration should be a problem that you solve once and by establishing open standards this should be achievable.
Please note - I am not advocating to add a lot of overhead to managing data, but merely a number of measures that should be taken into consideration for every data item that is shared across systems. The basics are as follows:
- First of all: Know what is should be shared - don't put measures in place on items that are unique to one system (unless that data is critical)
- Ensure integration of meta data with data - using standard XML schemas is very good approach
- Put in place some quality indicators - automated measures checking completeness and consistency of the data - this can be part of the meta data
- Also add other tags like time stamps and userids of people who have done something with the data (audit trail)
A controlled data management environment is a must for making business processes work across the company and across systems. The more you ensure the data adheres to transportable standards and is available with contextual meta data, the more you can integrate. At the end of the day data is only part of the information infrastructure. The real value is in what you do with it.
It also can help reducing the controls on document management. If documents can be automatically tagged via extracting words listed in master reference data, than that reduces the overhead in document management.
Labels: master reference data, meta data, semantic web, XML
Middleware
Since the day we started moving from two-tier to multi-tier IT architectures we have seen a growing complexity around how we integrate applications, data, etc. Through this we see a growing number of 'moving parts' emerging in every implementation stack. It increases flexibility, once up and running it will make deployment of changes a lot easier, it even helps with scalability, cost control and ..., but it becomes pretty complex.
The so-called
middleware market (covering hubs, connectors, real-time services, etc.) has been growing a lot and quite often I see that we need integrators to integrate the integrators (just to add to the complexity) and that's why it is important to really work on you architecture.
The rules are actually quite simple (and very old fashioned)
- Keep it simple (and this is rule nr 2 and 3 as well)
- Standardise - try to have one technology for the messaging and data transformation
- Do portfolio management (limit versions, vendors, etc.)
- And continue with keeping the overview of how data flows from one place to the other. This will help with challenging if another point to point interface is needed (why not channel it via a hub?)
- Always challenge the next middleware technology, because quite often there are simpler solutions possible (e.g. if you can solve it with data integration in the database, why have a messaging hub?)
- And if you have to choose a technology, than choose an open standard using XML, because that will keep your options open. Especially for integration with the outside world there is a lot of merit of using XML, since it is self-describing.
If you don't do these things you will see spiralling costs, the need for consultants with exotic skills and at the end of the day an unmanageable architecture ...
Labels: architecture, middleware, XML
The Value of Meta Data
In my previous post I already mentioned that I created a wiki dealing with meta data (check out:
http://scratchpad.wikia.com/wiki/MetaPedia) and the question is of course if this will ever evolve into anything. My experience with meta data (or information about information) is that it is a bit like flogging a dead horse. Still intuitively I think that if can add a lot of value.
So why is Meta Data management valuable?
- Meta data can help a lot when searching for documents. Adding meta data in web pages, documents, etc. increases the accuracy of the search result. The more meta data tags are standardised the more likely the users use them to their benefit
- Good meta data adds meaning to data, through context and description it is more likely data is understood and becomes information. An idea we tried out is to use something like a wiki as an online help - moving meta data into the knowledge management space
- Meta data can help managing information in a large enterprise context. When companies have to deal with many systems and complex data integration issues, than it is good to have an overview of data definitions, rules for data quality, rules of data integration (e.g. master source), etc.
- Meta data help automated data integration. When information is tagged using standard XML schema's than it becomes possible to integrate in all sorts of ways. Standardising on XML requires good meta data management and clarity on definition, else it won't work. The creator of the world wide web - Tim Berners-Lee - is now working on the Semantic Web and this all links back to having a standardised way of defining content (maybe more on this in a later Blog entry ...)
So if there is so much value why don't we do it?
Like said earlier - meta data management is a bit of a dead subject for most. People get excited by new gadgets, functionality, look & feel, but not about something administrative like meta data. It is like plumbing - nobody is interested where shit goes, but it is nice the sewer exists! Good meta data management is hard work and is not for people that are not very organised ... and without getting clear & quick reward nobody will do it.
So how to get it done?
- Make it exciting - this happened a bit when XML became a hype at the turn of the century. And the good news is - XML is now everywhere. So with other words - if you use the right 'hype' word, then it may get exciting. XML sounds like it is new, high tech, the next big thing and then some people actually start doing it
- Do it integrated - meta data management should not be standing on its own, like an ugly girl at a party waiting to be asked for a dance. When meta data is integrated with the data (like in XML), or with the application (like an online help), or is part of the companies knowledge management (like in Wiki's), then there is a change it will work out.
- Make it easy - if I need to tag this blog, than I get automatically a list of Tags I have used before. So why not have something like that built in part and parcel of tools like Word and Powerpoint? Maybe a future 'Vista' will do this automatically for us?
- Make it rewarding - as a manager you can reward people for boring things (like organising the junk-free day), so why not have reward for collecting & managing meta data?
Anyway - let's make meta data work - and if you happen to have more information on meta data, than add this to MetaPedia on Scratchpad Wiki.
Labels: meta data, search, semantic web, tagging, XML
