The Storage Architect » Atmos

Atmos Online Goes Offline

Chris M Evans — Thu, 01 Jul 2010 21:37:23 +0000

Chris Mellor at The Register picked up on the announcement from EMC on June 29th that the Atmos Online service is being withdrawn from production service and will return to a development platform. In the future, Atmos Online will only be available from partners, however the other flavours of Atmos (virtual appliance and hardware) will still be available.

There are plenty of rumours circulating on Twitter (mainly from Netapp employees) speculating on the reasons why this move has been made. Alex McDonald takes the route of making a Stalinist comparison which does seem slightly over the top, however his job is to be critical of the competition, so this shouldn’t come as too much of a surprise. The main thoughts are that either the service wasn’t making money or there was dessention from partners who also provide Atmos and so EMC were forced to pull their own service to placate their partners.

At the end of the day, only EMC know the answer.

Personally, I’m disappointed to see the service go. I did some analysis on the API when it was first released and I could see some benefits over both Nirvanix and Amazon’s cloud storage services.

However some questions still remain unanswered. Hopefully EMC may choose to answer them.

According to StorageZilla, Mozy is a big Atmos user. Does this mean that Mozy will be pushing all their data to be managed through one of their partners or will they retain their own Atmos infrastructure? If they choose to retain, why would it have been a problem to continue offering Atmos Online if the infrastructure was already there?
What plans are there for partners to offer Atmos services in all geographical areas? Here’s a link to the woefully poor AT&T Storage as a Service offering. It gives no real detail on what the service is, how to use it, where it is available and so on. Compare and contrast to Amazon AWS, where I can get full details, documentation and sign up immediately with an Amazon account and a credit card.

I’m not sure relying on partners like AT&T is the best move in the world. Amazon (and Nirvanix) features and functionality far exceed the Atmos offering. For once, EMC isn’t the lead player. They are going to have to work hard to keep up and the move away from offering the Atmos service directly doesn’t allow them to retain control and only looks like a backward move. I think this means we may be seeing Atmos’ slow death, despite EMC’s protestations otherwise.

EMC World 2009: Day 1 – Is DMX The Last Monolithic Array?

Chris M Evans — Mon, 18 May 2009 20:51:29 +0000

As I sit here at the start of EMC World Day 1, I’m pondering over some of the conversations of last night. The direction EMC are taking with V-Max, the Atmos product and Clariion makes me wonder if DMX could be classed as the last of the (EMC) monolithic storage arrays.

So, here’s the thinking. DMX arrays started to use the DAE (Disk Array Enclosure), previously deployed on Clariion. Atmos uses this as well. So effectively these three devices all use a common disk shelf technology. With the release of V-Max, EMC have moved away from the monolithic design of DMX and to a more modular, node-based controller architecture using Intel processors. So other than software, doesn’t that make all three storage arrays effectively the same product?

It’s About The Software Stupid

Perhaps EMC are making good on their promise of being a software company. Make the hardware a commodity and put all the investment into the software. After all, there’s no margin in hardware any more, or so we’re told.

Here’s another thing to ponder. If V-Max is going to be node-based, do all of those nodes have to be running Enginuity? How easy would it be to flip some of them into Atmos mode, Clariion mode, or even turn them into a virtual tape library? Perhaps you wouldn’t do that within a local cluster, but the option is there (and the intention from what EMC are implying) to move to geographically dispersed clusters.

With the V-Max architecture, all of a sudden the opportunities open up.

Enterprise Computing: EMC World Is (Almost) Here!

Chris M Evans — Wed, 13 May 2009 11:28:38 +0000

Yes folks, if you’re an EMC customer and unless you’ve been living under a rock for the last few months, you’ll know that EMC World starts next week. I’ll be attending for the first time (under the guise of amateur blogger).

If you’re going, feel free to drop me a mail/voicemail/Twitter DM if you want to chat. I’d be very interested to talk to anyone (especially customers) with views on where EMC is headed with their major platforms, including V-Max and Atmos.

Enterprise Computing: DMX-4 Is Dead – Long Live DMX-5!

Chris M Evans — Mon, 09 Feb 2009 20:49:15 +0000

Chris Mellor posted a great article recently, speculating on the future of certain storage products. He hinted at a new high end array from EMC, presumably an extension to the DMX range – tentatively titled DMX-5.

We’re about due for a new hardware release. Looking back, the schedule has been:

DMX-4 – 16th July 2007
DMX-3 – 25th July 2005
DMX-2 – 9th February 2004
DMX – 3rd February 2003
Symm 5.5 – 10th September 2001
Symm 5 – April 2000

So, expect a new array announcement around June/July this year. I’ve heard rumours of customers with secret EMC arrays, however they could have turned out to be Atmos and therefore just a bit of hot air. Alternatively, they may have been the new DMX-5 – although I’ve not got any proof to back that theory up.

Will EMC retain the DMX/Symmetrix heritage or will they take the opportunity to leapfrog the competition with something new?

The current DMX range suffers from a number of issues. The underlying architecture uses the concept of hypers to create LUNs, so RAID is based on creating LUNs from discrete slices of a disk.

Problem 1: As hypers are used to create LUNs, a LUN will consist of a maximum of eight hypers across 8 disks using RAID-5 (7+1). Creating these LUNs means making a hyper of a size which is divisible by the intended LUN size, so if you want a LUN of 70GB on RAID-5 7+1, then you need to create eight 10GB hypers. If you want to stripe across more devices, then you have to go with meta-devices, which concatenate more than one LUN together. This means either bigger LUNs or carving smaller hypers. Typically though, an array configuration will be pre-determined at installation time and making changes to hyper sizes can be a torturous process, which is usually avoided.

Problem 2: Legacy Code. Symmetrix and DMX all worked off the concept of physical mirrors of a logical LUN – commonly known as mirror positions. A BCV and an SRDF device become a mirror position, with a maximum of four permitted. Without me offending the excellent programmers who have crafted the Enginuity microcode, EMC have had to effectively shoehorn new features in – for example Snaps & Clones to replace BCVs and Thin Provisioning. The inherent Symmetrix/DMX design doesn’t lend itself to the implementation of new features.

So, will EMC make a break and go for a complete redesign for DMX-5? Wide striping seems essential, as does dynamic LUN creation, however top of my wish list is the ability to move data between tiers of storage within the same array – and for this to be granular enough to make it useful.

This requirement leads on to full policy-based placement of data and automated migration between tiers to maintain the service level requirements of the policy. In turn, this leads to integration with Atmos, providing a full end-to-end policy-based data placement and management. Wouldn’t that be good!

Decho – EMC Takes Over The World

Chris M Evans — Mon, 17 Nov 2008 13:00:00 +0000

Chris Mellor just announced the news that EMC have bundled their Pi and Mozy acquisitions into a single entity, branded as Decho. I was far too slow and Storagezilla beat me to the mandatory EMC post.

So, with Mozy and Pi we now have our data and backups online in the EMC cloud – which conveniently arrived last week as Atmos.

I may have been somewhat overly negative towards EMC in previous posts (they’re big boys, I’m sure they can take it), however the layering of cloud storage offerings with Atmos as the foundation (assuming they eat their own dog food and use it) and content/backup over the top does move EMC into a new and interesting market segment in offering storage services rather than just tin (or software for that matter).

Where’s the logical conclusion as to where EMC are headed? Is the move to Storage-as-a-Service an implicit acceptance that, over time, hardware will become even more commoditised and that services are the future? In the long term, surely that’s the ideal scenario for the end user; all data and services in “the cloud” somewhere with no need to know where/how the data is stored other than service level and performance guarantees. It’s not likely to happen in the near future but as a long term trend, it is certainly compelling.

Obligatory Atmos Post

Chris M Evans — Thu, 13 Nov 2008 07:28:00 +0000

I feel drawn to post on the details of Atmos and give my opinion whether it is good, bad, innovative or not. However there’s one small problem. Normally I comment on things that I’ve touched – installed/used/configured/broken etc, but Atmos doesn’t fit this model so my comments are based on the marketing information EMC have provided to date. Unfortunately the devil is in the detail and without the ability to “kick the tyres”, so to speak, my opinions can only be limited and somewhat biased by the information I have. Nevertheless, let’s have a go.

Hardware

From a hardware perspective, there’s nothing radical here. Drives are all SATA-II 7.2K 1TB capacity. This is the same as the much maligned IBM/XIV Nextra, which also only offers one drive size (I seem to remember EMC a while back picking this up as an issue with XIV). In terms of density, the highest configuration (WS1-360) offers 360 drives in a single 44U rack. Compare this with Copan which provides up to 896 drives maximum (although you’re not restricted to this size).

To quote Storagezilla: “There are no LUNs. There is no RAID. “ so exactly how is data stored on disk? What methods are deployed for ensuring data is not lost due to a physical issue? What is the storage overhead of that deployment?

Steve Todd tells us:

“Atmos contains five “built-in” policies that can be attached to content:

Replication
Compression
Spin-down
Object de-dup
Versioning

When any of these policies are attached to Atmos, COS techniques are used to automatically move the content around the globe to the locations that provide those services.”

So, does that mean Atmos is relying on replication of data to another node as a replacement for hardware protection? I would feel mighty uncomfortable to think I needed to wait for data to replicate before I had some form of hardware-based redundancy – even XIV has that. Worse still, do I need to buy at least 2 arrays to guarantee data protection?

Front-end connectivity is all IP based, which presumably includes replication too, although there are no details of replication port counts or even IP port counts, other than the indication of 10Gb availability, if required.

One feature quoted on all the literature is Spin Down. Presumably this means spinning down drives to reduce power consumption; but spin down depends on data layout. There are two issues; if you’ve designed your system for performance, data from a single file may be spread across many spindles. How do you spin down drives when they all potentially contain active data? If you’ve laid out data on single drives, then you need to move all the inactive data to specific spindles to spin them down – that means putting the active data on a smaller number of spindles – impacting performance and redundancy in the case of a disk failure. The way in which Atmos does its data layout is something you should know – because if Barry is right, then his XIV issue could equally apply to Atmos too.

So to summarise, there’s nothing radical in the hardware at all. It’s all commodity-type hardware – just big quantities of storage. Obviously this is by design and perhaps it’s a good thing as unstructured data doesn’t need performance. Certainly as quoted by ‘zilla, the aim was to provide large volumes of low cost storage and compared to the competition, Atmos does an average job of that.

Software

This is where things get more interesting and to be fair, the EMC message is that this is a software play. Here are some of the highlights;

Unified Namespace

To quote ‘zilla again:

“There is a unified namespace. Atmos operates not on individual information silos but as a single repository regardless of how many Petabytes containing how many billions of objects are in use spread across whatever number of locations available to who knows how many users.”

I’ve highlighted a few words here because I think this quote is interesting; the implication is that there is no impact on the volume of data or its geographical dispersion. If that’s the case (a) how big is this metadata repository (b) how can I replicate it (c) how can I trust that it is concurrent and accurate in each location.

I agree that a unified name space is essential, however there are already plenty of implementations of this technology out there, so what’s new with the Atmos version? I would want to really test the premise that EMC can provide a concurrent, consistent name space across the globe without significant performance or capacity impact.

Metadata & Policies

It is true that the major hassle with unstructured data is the ability to manage it using metadata based policies and this feature of Atmos is a good thing. What’s not clear to me is where this metadata comes from. I can get plenty of metadata today from my unstructured data; file name, file type, size, creation date, last accessed, file extension and so on. There are plenty of products on the market today which can apply rules and policies based on this metadata, however to do anything useful, then more detailed metadata is needed. Presumably this is what the statement from Steve means: “COS also implies that rich metadata glues everything together”. But where does this rich metadata come from? Centera effectively required programming their API and that’s where REST/SOAP would come in with Atmos. Unfortunately unless there’s a good method for creating the rich metadata, then Atmos is no better than the other unstructured data technology out there. To quote Steve again:

“Rich metadata in the form of policies is the special sauce behind Atmos and is the reason for the creation of a new class of storage system.”

Yes, it sure is, but where is this going to come from?

Finally, let’s talk again about some of the built-in policies Atmos has:

Replication
Compression
Spin-down
Object de-dup
Versioning

All of these exist in other products and are not innovative. However extending policies is more interesting; although I suspect this is not a unique feature either.

On reflection I may be being a little harse on Atmos, however EMC have stated that Atmos represents a new paradigm in the storage of data. If you make a claim like that, then you need to back it up. So, still to be answered;

What resiliency is there to cope with component (i.e HDD) failure?
What is the real throughput for replication between nodes?
Where is the metadata stored and how is it kept concurrent?
Where is the rich metadata going to come from?

Oh, and I’d be happy to kick the tyres if the offer was made.