Architecture in different dimensions
One of the mistakes I see on a regular basis is that designers try to fit too many different dimensions in one architectural design. Therefore in this post I would like to plead for a few standard views that describe the architecture in different dimensions.
In my view I would plead for a high level data flow as the basis of any data architecture. In this flow you can see the main data stores and interfaces. The data stores should fit in the data architecture patterns described a few weeks ago. The data flow can be divided in the main layers for e.g. corporate / master data, project data / data marts, archives, etc.
On top of this view you can have multiple solution architecture views for the technology stack related to a solution. This should include the main data stores at the bottom (in line with the main high level data flow), middleware in the middle and applications + viewers at the top.
Providing these different views will help the discussions on data flows (e.g. where is the master?) and the actual technical implementation of a particular solution.
Labels: architecture, master reference data, meta data, middleware
De-coupling data from its usage
One of the trends in integration architecture at the moment is the drive for a Service Oriented Architecture (SOA). For years we have been trying to rationalise applications and databases and time and time again we realised that with every migration we had to migrate a lot other stuff as well (like the increasing number of interfaces). Moving one application to something else would also lead to moving a lot of interfaces, reports, dashboards, you name it.
So now the idea is to hide all the complexity behind an abstraction layer. Why not publish all information via Web services, so everything below this becomes easy to replace with something else? 'All' we need to do is to describe the way want to see the information. The best way to do this is via defining a flexible data model (through an XML schema) and this is where the new trouble starts, because haven't we tried to do this before at the database level with enterprise data models? The good news is that this time we have more chance to succeed, since the middleware layer is much more flexible, since the data is not stored in the format of the schema, it is just described that way. And when needed you may even have more than one schema decribing the same information, without the need to store the data in a different way.
It looks promising, but we have a long way to go, since we only have started ...
Labels: architecture, SOA, web services, XML
Architecture Patterns
Data architecture is not easy if the nature of the data stores in the architecture is not understood. Usually it is good to split stores with different functionality into different architectural elements; and quite often this choice is driven by the nature of the data (transactions vs reference data). A few examples of rules are below ...
- Spit reference data management from managing transactions
- Split transaction data stores from business intelligence solutions like warehouses and data marts
- Split project data stores (with data that can be changed and reinterpreted) from corporate data stores where final data resides
- Split current data from archived data
Looking at these elements it is possible to start seeing patterns; e.g. the pattern around a transaction data store where the master reference data is managed separately, the BI is arranged through data marts and retention is arranged in an archive store. Creating these patterns for specific business situations helps with understanding if proposed solutions are complete, or if designers plan to cover requirements with the wrong type of store (if you have a hammer, everything looks like a nail).
Labels: architecture, master reference data, principles
Transaction data versus reference data
One of the key steps in data architecture is the understanding of the difference between Transaction data and Reference data.
- Transaction data is about events, actions, state changes and are usually described with verbs. Think work management, thinking paying the bills.
- Reference data is about things, physical or imaginary and are usually described with nouns. Think assets, cost centers, people.
When 'architecting' solutions we see that transaction data is treated differently than reference data.
- Transactions usually form a flow of events and are therefore characterising the change through the lifecycle of a business process. Further transactions can be summarised, aggregated and become management information, or archived when not longer needed. A typical transaction environment is SAP
- Reference data usually relate to one or more transactions and can be same all over the business process (although transactions may change the attributes of a reference object). Reference data can be organised in hierarchies and then the higher levels of the hierarchies are the aggregation levels for the transaction data. Reference data shared across a number of functions can be called master reference data. In order to keep this data consistent it is important to share it from one place (enter only once, use many times)
This all seems pretty obvious, but the experience I have is that people mix up the two types and do not understand that there are different rules governing the management of systems with this data.
Labels: master reference data, transaction data
Data Management as a Process
It is good to re-iterate that DM is a process. Quite often business users think that a one off exercise in cleaning up the data helps them to get things right ...
I would like to make the comparison with brushing your teeth - if you leave it for a day it does not become an issue, but if you leave it for a while it will start to hurt. Managing data is like any maintenance activity; it is recurring and people do not like it.
So plan the right resources, schedule the activities in conjunction with the business processes which are supported by the data, execute it with sufficient resources, and eventually evaluate the results and learn.
This can be done two ways: 1) as an ongoing activity within the business process - which requires quite often a behaviour change or 2) by a dedicated group DM professionals which helps building DM skills and professionalising the activity, but which is only possible with sufficient volume of work (and suffient standardisation of the activities)
Bottom line is that it should be understood that this is a job to be done - and it should be rewarded!
Labels: behaviour, Process, Skills
