To hype or not to hype
I have written a few times before about SOA (service oriented architecture) and recently I get more and more exposed to consultants in suits that have SOA in all their Powerpoint slides and I start to realise that we are at the top of the hype cycle with respect to SOA and data integration. Quite often it becomes something that sounds like this: 'if you have a hammer than everything looks like a nail'.
So my advice to all people that require integration:
Avoid these people and concentrate on the question: what kind of integration problem am I trying to solve!My view is that we should sell real solutions and not label everything with the next buzzword. Note that I do not disagree with the general principles of SOA - I'm just very sceptical about overselling something that is rarely understood and which is still hard work! And the other thing is: Business people do not want to hear about SOA or any other techno jargon. They just want to see solutions that work and do not want to sink millions in mega projects with a high chance of failure.
So - consultants - let's get back to the coalface and show me an incremental integration strategy that delivers something quickly! And forget about SOA.
Labels: SOA
More on integration ...
As I wrote before, one of the main aims of Data Architecture is to support the needs for integration in the organisation. There are different ways to achieve this and therefore Architects are usually in favour of Big Projects. Developments like a Common Database (with a common data model) or a Data Services layer (with a canonical model in the service layer) are the most common projects. These projects are never popular and hardly ever a success.
At the end of the day most organisation live with a lot of point-to-point connections and because they are more practical, cheaper & faster to build they may be a better solution (even though we always way that they are less maintainable). So the challenge of the architect is to balance the cheap, fast ad hoc link with a nicer layered architecture. Usually the cheap thing wins and for a good reason ...
One way of achieving a solution for this is balancing act is to define a standard for the common framework and let projects develop their 'point-to-point' as part of this standard. Through publishing the specific link as meta data (with an XML description) we will see the services layer grow!
Labels: architecture, integration, middleware, SOA, XML
Application Services
Probably the big paradigm shift currently happening in the industry is the move to 'Services' hosted by 3rd parties. Who wrote some years ago already that the '
network is the computer'? (well, it was one of Microsofts rivals - SUN). This vision is becoming a reality now. Why have Word and Outlook installed, if their online sisters are more reliable, cheaper, ...
Right now Google apps is growing to become a serious competitor of Microsoft, and even Amazon is entering this market with offering storage services ... Soon we have SAP moving to a service model (they are already working on this for some years) and soon after this others will follow.
This trend will give new opportunities and challenges to the industry. The opportunity is that we get the option to focus more on what really matters for the company (because services like storage, word processing, accounting are becoming commidities, just like electric power ...). The other challenge is that we need to rethink the way we manage our networks. At the end of the day the only asset that stays is 'our' information (the data and documents) that need to be managed, protected, etc. Opening up to market services changes this model.
I see it mostly as an opportunity, since it will put information management firm at the center of what we have to manage!
Labels: Google, Information Management, SOA
Architecture and Information Services
I strongly believe that most organisations today are mostly dealing in information and therefore - building a layer of 'information services' is something that should be considered when trying to build a sustainable and agile architecture.
What is the story of information services? Information services are a way of decoupling data from functionality. In the past this was already done by creating databases - but usually the databases were application specific. In the mind of most programmers there cannot be data without an application.
When applications started to grow and started overlapping, then the concept of shared data emerged - this happened at the end of the 'eighties' and suddenly people started to think about designing major data models. One of the most glorious attempts was the 'Epicentre' datamodel - designed by the Petrotechnical Open Software Corporation, an object relational database for the oil industry with more than 2000 tables. It was very clever, very flexible and very hard to implement. And therefore it was never a success.
Obviously these heroic attempts were doomed and with the growth of off-the-shelf applications we saw the rise of ETL tools (to get a grip on all the interfaces) And quickly after the rise of ETL, with the rise of the Internet, we saw a complete market growing of middleware solutions.
As information managers found out - it is also possible to create spagghetti with middleware solutions as well and now eight years after the millenium we are back at where we started - we are crying out to get more control on the information.
And this is where the information services story comes back again. If you put information at the center and 'wrap this up' as a service, using a canonical data model (a bit like the Epicentre data model, but then more pragmatic ...), then it is possible to reduce the complexity of your architecture. The data can be in various places (so don't think about large corporate data stores) and the middleware just 'knows' where to get it. The applications just 'shop' for data from the middle layer and the 'bus' is doing the work. Well this is where the complexity is - building the 'bus' is the hard work - since it requires
- rationalisation of data sources (agree where is the master)
- common definitions (or at least agree how to transform)
- routing (agree how the data flows)
And this is where we in IT are not good at - it is great to define a lot of new functionality, but to crawl through the sewers of data management is very hard. But if you want to succeed in getting control on your information centric entreprise, then think 'information services'!
Labels: architecture, middleware, SOA
Master reference data management
Implementing a service oriented architecture (SOA) also leads to the move to set up master reference data management services.
I am sure that there are no real best practices around at the moment, so therefore this is a bit like exploring undiscovered territory. The big step is that you start to de-couple data from functionality. With other words - the management of data becomes a business process in itself.
So how do we get this done? Some people start straight away with developing the best possible data model, but I think that this is not a good idea. Don't develop a data model that would meet everybody's requirements from the start, but follow the steps, as laid out earlier in ...
http://infomgr.blogspot.com/2007/06/data-management-fundamentals.html and the best is to start small. So in short do the following:
- Pin down the scope & keep this as small as possible to start
- Define the ownership, including the operational responsibilitie
- Define the process / data flow, including the operational roles
- Set up the data model & keep this as flexible
- Finally - set up the 'service'
The service itself should contain the following:
- The canonical data model + optionally the container for the data (and keep the scope limited to start!)
- A messaging service; to transport the ins/upd/del transactions
- A routing service; to move the data to the right place
- A key mapping service; to know how to translate keys, if required
- And optionally a user interface for power users
This all looks a bit daunting, but by starting small it is a feasible feat. Let's see if I can convince my colleagues of this wisdom as well!
Labels: master reference data, SOA
Canonical Data Model
The main challenge continues to be the data model. In SOA speak: the canonical data model. Through some sort of difficult reason it is very hard for most IT people to deal with data models. Obviously they are not so sexy as the functionality in the application. Unfortunately it still requires quite a bit of skill to develop one ... and expecially an intermediate data model needs to fit a lot of requirements. It needs to...
- Support the superset of requirements of the related systems - and therefore it is important to model cardinalities, identifiers, attributes, etc. in the most flexible way. Cardinalities need to be flexible (by default many to many). Identifiers need to be unique (by default meaningless numbers). And with every attribute it needs to be clear that this is an attribute and not potentially an entity ...
- Support the reference data over time (4D), i.e. everything needs to be time stamped and no reference data gets deleted ...
If developers want to meet all these requirements, than they need to dust off some skills we dropped during the nineties.
Labels: meta data, SOA, XML
Enterprise Service Bus
The flavour of today in integration architecture is the Enterprise Service Bus (ESB), which is in a way nothing new. The idea of integration via an intermediate model has been around for some time and has led to many disappointments. But maybe this time there is a chance to make it work, since technology has moved on.
So what's new? First of all XML has matured to some sort of lingua franca; gradually all players in the market start to support the paradigms of SOAP, XML schema's, message based services, etc. And that means that momentum is building up. Second - the main difference is that everything is loosely coupled and therefore the responsibility for integration is now firmly put at where it belongs - with the applications. With other words - it is not the middle layer that needs to take care of everything, and this may be a factor determining its success.
Labels: architecture, SOA, XML
De-coupling data from its usage
One of the trends in integration architecture at the moment is the drive for a Service Oriented Architecture (SOA). For years we have been trying to rationalise applications and databases and time and time again we realised that with every migration we had to migrate a lot other stuff as well (like the increasing number of interfaces). Moving one application to something else would also lead to moving a lot of interfaces, reports, dashboards, you name it.
So now the idea is to hide all the complexity behind an abstraction layer. Why not publish all information via Web services, so everything below this becomes easy to replace with something else? 'All' we need to do is to describe the way want to see the information. The best way to do this is via defining a flexible data model (through an XML schema) and this is where the new trouble starts, because haven't we tried to do this before at the database level with enterprise data models? The good news is that this time we have more chance to succeed, since the middleware layer is much more flexible, since the data is not stored in the format of the schema, it is just described that way. And when needed you may even have more than one schema decribing the same information, without the need to store the data in a different way.
It looks promising, but we have a long way to go, since we only have started ...
Labels: architecture, SOA, web services, XML
