We’re [...] ]]> A quick Tweet with Chris Mellor just reminded me of something I touched on a few months ago but always meant to write about in more detail.  It’s a semi-serious analogy (it is Friday after all) but there’s a hint of the possible about it, so here goes.

We’re all creating too much content.  Whether that’s at home or at work, there’s just too much stuff about.  It’s duplicated files, downloads of software, documents we created and never looked at, pictures (some useful, some we’ll never print but just retain regardless).  What we need is a waste dump for all of this infrequently referenced content.

So, the cloud storage offerings become an extension of your desktop.  A background task farms through your files, looking for stuff you’ve not referenced for a while, pushing it off to a cloud storage vendor and leaving a stub or link to the file in case you need it.  They analyse the file, determine whether the content is useful and push it back to you if it is, delete it if not (alternatively you get a report indicating whether your data is any use – you choose to delete or retain at low cost in the cloud).  All of the functionality is policy-based, set by the customer.

What’s in it for you?  As a home user, you get your pictures, word documents, video, MP3s all nicely indexed and managed.  You don’t have to think about the process of keeping stuff organised, just throw it at your virtual recycling centre.  If you’re a business, your files gets scanned for illegal or copyright content; it gets checked for files which meet certain policies on business-related content – it could even be farmed for the content, depending on what the files are.  As a business, you keep your onsite storage costs down, you get offsite backup & location independence and the ability to certify to the authorities that you pro-actively seek out and remove content which may result in litigation from copyright owners.

What does the vendor get?  Well, customers may choose to backup and/or archive their files – for a fee.  We’re all inherently lazy and if someone else confirms that you’re only backing up what’s necessary then you’ll be attracted to the service.  Businesses can be charged for processing content and the cost made more attractive than simply keeping more data on bigger and bigger filers.  Bear in mind that cloud vendors are going to have to differentiate themselves somehow.

Personally I know I’m guilty [...] ]]> In a recent post, Hu Yoshida refrences an IDC presentation discussing the rate of growth of structured versus unstructured data. It seems that we can expect unstructured data to grow at a rate of some 63.7% annually. I wonder what actual percentage of this data represents useful information?

Personally I know I’m guilty of data untidiness. I have a business file server on which I heap more data on a regular basis. Some of it is easy to structure; Excel and Word documents usually get named with something meaningful. Other stuff is less tangible. I download and evaluate a lot of software and end up with dozens (if not hundreds) of executables, msi and zip files, most of which are cryptically named by their providers.

Now the (personal) answer is to be more organised. Every time I download something, I could store it in a new structured folder. However life isn’t that simple. I’m on the move a lot and may download something at an Internet cafe or elsewhere where I’m offline from my main server. Whilst I use offline folders and synch a lot of data, I don’t want to synch my entire server filesystem. The alternative is to create a local image of my server folders and copy data over on a regular basis, trouble is, that’s just too tedious and when I have oodles of storage space, why should I bother wasting my time? There will of course come a time when I have to act. I will need to upgrade to bigger or more drives and I will have (more) issues with backup.

How much of the unstructured data growth out there occurs for the same issues? I think most of it. I can’t believe we are really creating real useful content at a rate of 63.7% per year. I think we’re creating a lot of garbage that people are too scared to delete and can’t filter adequately using existing tools.

OK, there are things out there to smooth over the cracks and partially address the issues. We “archive”, “dedupe”, “tier” but essentially we don’t *delete*. I think if many more organisations operated a strict Delete Policy on certain types of data after a fixed non-access time, then we would all go a long way to cutting the 63.7% down to a more manageable figure.

Note to self: spend 1 hour a week tidying up my file systems and taking out the trash…..