Unbelievable
Last month I wrote about a cyclone in Oman and the one thing that struck me after the event was the lack of preparedness everywhere. I think the worst hit was the Royal Oman Police. I guess they has their data center in the middle of the floodplain where some of the worst damage happened and because of this they lost all their main servers. They clearly had no business continuity plan (BCP) and no disaster recovery (DRP) in place and did not think of a scenario of 3 feet water in the main server room.
Until the cyclone, Oman looked like leaping into the twentyfirst century with their advanced goverment IT integrating the civil administrations of people, immigration, drivers licenses, work permits, car registration, etc. etc. I was amazed by the advanced way in which the data was integrated (e.g. my drivers license number was the same as my number of my id card which allowed me to work in the country, and my car was linked to my drivers license - and therefore also the traffic offences ...). This clearly added a lot to the efficiency of the country, but the bottleneck was the data center in the middle of a wadi.
After the cyclone I could not export my car (since it was not known if there were traffic offences still outstanding), immigration moved back to a paper based system and all kinds of other services grinded to a halt. Luckily Oman is part of the Arab world and therefore there is always a way to get things done (the secretary of the main officer in charge was a cousin of the guy helping me, etc.) and therefore I could leave the country in time without major issues! No data management can help with that
So bottom line is - I think what Oman achieved before the cyclone in terms of government data management can be seen as a 'best practice' (just try to compare this to the UK where hardly anything is linked!), but when leaping forward we cannot forget the basics - arrange sufficient protection for the vital records and have a plan in place for the worst!
Labels: business continuity, disaster recovery
Vital records protection
Due to the cyclone of last week I have had quite some thoughts on DRP and BCP, and concluded that the scope of this should be pretty limited. Risk management is a good way of scoping (only records unavailability that causes a high risk needs to be considered as part of the scope), but this can be a slightly too narrow approach and may lead to short term gains and longer term problems. Therefore it is important to define for a company the
vital records. A simple definition is: Without these records you can close the shop. So what are the main parameters in defining vital records?
- Legal: By law you need to have these records - usually these records are related to agreements, contracts, financial transactions and people. Quite often these records have a legal retention date (think Sarbanes-Oxley act). Obviously these records need to be properly protected. The most important (and therefore vital documents) are the ones related to major agreements. The bulk type (invoices) need less protection.
- Asset Integrity: Records on the structure and state of maintenance (integrity) of facilities are vital for any operation. Think drawings, designs, inspections. Quite often these records need to be retained indefinitely and they require a high level of protection in terms of protection against damage or loss. Quite often they're not confidential.
- Confidentiality: Some records in the company are company secrets that can be the differentiator for doing business (think intellectual property, major strategic documents). These records need protection both from a confidentiality and a physical protection perspective.
So what to do? I think first of all it is important to have records in an open digital format, i.e. stored on a computer and easy to open with normal desktop tools (think PDF). Paper is great to work with, but digital is the only proper backup mechanism. So if you have paper than it is important to have at least good quality scans of all vital records. The scans need to be indexed properly and need to be made available online (with sufficient security of course). Obviously some records (like facility drawings) may have a more specific format (e.g. Autocad).
Further it is important to have the digital records protected physically against any disaster (flooding, storm, fire, ...) and the easiest way to establish this is having them stored in more than one place (preferably more than two). Note that these places need to be geographically apart. In some countries companies have decided to have their information backup abroad (especially recommended for more volatile places). So if you have your records in Amsterdam (or New Orleans), than it is wise to have a backup in place in a higher place (so not next door, but say in the Alps or the Rocky Mountains).
The backups have to be made on a regular basis (daily) and more importantly: you have to test if the backup can be restored! Note that the same security measures (on accessibility) need to be in place on the backup (so confidential data is also protected in the backup site).
So bottom line: it is important to know what is vital for your business and it is important to have good protection in place for them. Not just in terms of security, but also in terms of physical protection against disasters and the best protection for the latter is replication.
Labels: business continuity, disaster recovery, retention, Risk Management, security
Managing Risks and the world of Web 2.0
Quite some time ago I wrote about Information Management that it is actually all about managing risks. And our little cyclone has reminded me of this fact. We suddenly realised that most information in the company is actually not important (or let me rephrase, not important enough for business continuity)
So a lot of what we do in Information Management is more at the 'nice to have' end of the scale of things. But having written this statement does not mean that there are some nice things about IM / Web 2.0 that are now emerging as elements that can add value in the continuity of a business or be of help during recovery from a disaster.
The Web2.0 has brought us wiki's, blogs, ... and this is how they could help:
- Wiki's can lead to more up to date procedural information, but can also help with collecting the learnings from a disaster event. Everybody can contribute! Many people know more than a few isolated auditors
- Blogs can help during the disaster to share news and bring people up to date on the status of services and other things. I noticed for instance that AP, the news agency has started to use pictures from Bloggers in their news coverage. Mobile blogging is also possible, so why not use your phone for things like this?
- The free online storage spaces for content (pictures, movies, but also documents etc) can act as a temporary business continuity site when your own services are down
But having these things playing a more vital role means that these Internet based services become more vital and therefore become part of the DRP/BCP (disaster recovery planning and business continuity planning). That's a way to become important!
So the infrastructure side of things needs a possible rethink as well. Usually data centers are setup as a single point of failure and having the Internet as a vital piece of communication technology does not allow for single points of failure. So setting up the companies infrastructure as a set of networked nodes is a model worth considering as well.
Labels: business continuity, disaster recovery, Risk Management, Web2.0
Some notes on Disasters
A few days ago we were hit by a cyclone, and this brought up the subject of disaster recovery and business continuity again. We have spent many hours discussing vital services in the past and how to ensure continuity for them (or how to ensure that we could recover from a disaster) and thanks to the cyclone we could test our assumptions.
On
disaster recovery (DR) I noticed that we were lucky. We had an orderly shutdown before the storm hit us and no real damage was done to the data center. So recovery was easy - it was just a matter of getting the computer floor dry, getting power up and running the start-up sequence.
Things would have been worse if the storm had wiped out our facility, since we had only limited backup equipment - and most data was kept in the same site (just 500 m down the road). But arranging a full mirror with sufficient distance is expensive ...
On
business continuity (BC) I noticed that really most services are not vital. The main office was completely deserted, but production continued. The next morning when I arrived in the data center I saw nobody around, but all services were humming happily. Is it really that most what we do is not really necessary? It was good to see that most production operations just need basic IT functions and therefore they were not hit. This also gives me some thoughts about our plans for IP telephony (IPT), since this is a more vulnrenable service that the good old fashioned system that we have today, so it is clear that we still need to have it as a back-up.
So what were the things I missed during the storm? I actually missed mostly my access to the Internet! Being cut-off from the rest of the world with just rumours - no real information - can be dangerous. People were making assumptions about the wind, the waves, etc. and through this had the chance of making the wrong decisions. So having a good news service up and running was vital. If the company also has IPT in place, than Internet + IPT become the most essential services to keep up. For the rest it is mainly about keeping power & water up running, so all the basic IT needed for this is vital. The rest can just run on a laptop (e.g. information on emergency and recovery procedures, or even on paper).
Bottom line is that for BC it is only the bare bones of the IT services that are important and this service obviously needs electricity, so any back-up system in place should be able to run for some time in a limited mode on maybe a small generator or batteries, so we keep the basic utilities up and running + the communication to the outside world. Anything else is luxury. And for DR I think it helps to have some conscious decisions on what information (+ apps + hardware) should be available in a recovery site at some distance, because you may need it one day ...
Labels: business continuity, disaster recovery
