Being an Information Manager...

Making Controls Work

Data architects need to balance constantly between what we want in terms of easy to use functionality and building in some level of control. Users do not want to be bothered with storing data in the right place, or adding all sorts of meta data, but data managers know that without this it is not easy to re-use the data elsewhere, or later during the life cycle of the data.

So one of the holy grails for data architects is to find that right balance between easy functionality and the controls. My view is that we need to focus with controls on the 'must-have' and further see how much we can already populate through some smart algorithms. At the end of the day we all know that if things are automated they will happen (like the a search index), and when you rely on the user it is more likely that meta data stays incomplete and inconsistent, unless there is a lot of control. The right balance is even easier to find if some of the user interaction can be supported by work done by automated processes - e.g. if the meta data can be pre-populated, and users only need to click OK (and take out the glaring mistakes), then this process may work.

Another option is to make the data quality more visible; more transparent retrieval mechanisms (like e.g. Spotfire or BusinessObjects) are great enablers for this. Suddenly users see that data is missing and may even grasp the effect of this.

Labels: behaviour, quality, retrieval

posted by Evert Ruijs @ 5:43 AM 0 comments

Data capture vs data retrieval

One old concept of data management which is still valid today (and at the same time insufficiently adopted) is the architectural split between data capture vs data retrieval. It seems a bit artificial to split datastores in this way, but in larger architectures it makes absolute sense. This is not only because of the traditional reasons of tuning the retrieval database for performance (the old data warehouse concept), but mainly because of lots of practical reasons. Here are some:

• Data capture is usually a complex process with steps for QC and validation, while retrieval is read-only. This leads to different data models, security models, etc.
• Data capture is usually for just a limited number of users with various access rights, while data retrieval requires a focus on sharing
• Data retrieval environments are focused on information retention over time
• Data capture environments should only exist once for a data type while data retrieval environments can exist in multiple ways. Once data is created it can be shared or replicated instantly via messaging services

Having this concept in the back of the mind whilst architecting a data environment for an enterprise (so not for small systems!) is very useful and can simplify enterprise wide solutions.

Labels: architecture, data capture, retrieval

posted by Evert Ruijs @ 1:02 PM 0 comments