Now, don’t forget I’m a fan of the concept of cloud storage, but I also think that we need better data availability models to cope with the failure of a single provider (I wrote a series of posts on this starting here).  However, reading the comments on the TechCrunch site, one entry from Chris Walker made me think that people aren’t understanding the way IT works:

“I believe data stores in S3 auto-replicates across multiple data centers so theoretically you should not encounter a data loss like this one – unless all of amazons data centers get nuked.”

Data loss is not purely a hardware failure issue.  Information is lost for many reasons; whether that’s malicious data destruction, lack of forethought on IT design, a rolling disaster, failure in replication, or something else, you can’t assume just because data exists in multiple locations that it will be 100% safe.  

Data security is as much about process as it is the technology.  Here’s a few questions to ask:

• Is my data replicated in real-time (so if I delete/corrupt it, I ruin all copies)?
• Is my data saved in snapshots (helps prevent the above issue)?
• Is my data secured on multiple technology platforms (e.g. primary, secondary and/or tape storage)?
• How is my data being replicated (tools and process)?
• How can I prove my data has been backed up/replicated successfully?
• What level of user/electronic/physical security is in place?

Unfortunately, the appeal of online storage services today is the low cost and ease of use.  Perhaps we should also be giving more weight and consideration to the value of our data before we commit to using these services; it may then be worth paying a little more to get a more resilient service, or to insure against the consequential losses of losing valuable data.

• An engineer was testing the Halon system in the newly completed computer room extension at my first site. Unfortunately he’d forgotten to turn the key to “test” before pressing the fire button and let off the Halon in the whole of the datacentre with both the equipment up and running and operators in the room mounting tapes. Needless to say, they were out of there like a rat up a drainpipe!
• During a recent delivery of storage arrays; one array literally fell of the back of the lorry. It had to be shipped back for repair…
• An array installation I managed in one site was mis-cabled by both the electricians and the vendor. When it was powered up, it exploded…
• On a delivery of equipment, the vendor arrived at the loading bay at the datacentre. As the loading bay door was opened, it jammed and broke, just too low for the arrays being delivered to be pushed under the door. The vendor had to return the following day after the broken door had been repaired.
• A tape drive on a StorageTek library I worked on took 12 hours and around 6 staff to complete. Half way through the upgrade, we took a go/no go point and checked both the MVS and VM installations to ensure the new drives worked. The MVS connected drives were fine; the VM drives had a “minor problem”, so we proceeded, in anticipation of resolving the VM problem. The following day we discovered the VM problem was not correctable and had to purchase additional drives at considerable cost.
• After loaning out some disk space to a “temporary” project, we had a hardware failure 3 months later. It turned out that the team had forgotten to ask for backups for their data and 3 months of the work of a dozen people was lost.

Fortunately, most of the above were not life threatening (except the first, which I was not involved in directly). However one of these problems did result in data loss (albeit on a development environment). It shows how many times the unexpected and unplanned can happen and mess up the best laid plans.

Care to share any of your stories?

There have been lots of data loss issues reported in the last year, mostly within large corporations. What amazes me is that systems are designed to allow a single individual to export so much sensitive data in one download and to be able to put this data onto a CD. Even if a “Junior Official” had broken the rules, its about time systems were designed to prevent data exports like this to happen in the first place.

Systems must be designed to be “closed loops” with data being exported only to other secure systems across secure links, rather than relying on sending CDs in the post. Our own government should be setting the standard for industry. Unfortunately this will just become another reason to mistrust the tax man.