Defining Unified

The word “unified” in Unified Storage is meant to mean a unification of the common access protocols; block and file. Typically, many arrays cover only a subset of available storage access methods – CIFS/NFS, iSCSI and/or Fibre Channel. However, take a look “under the hood” and you will see that many of these products – from a physical perspective – are not unified at all.  EMC’s VNX is a marketing triumph, bringing the CLARiiON and unloved Celerra platforms together into what is sold as a magical single product.  In truth it’s far from that, with the two separate products still existing in physically separate shelves, albeit with a shiny new bezel.  Having a single management interface, of course does help to complete the illusion.

Probably the only big vendor selling a true unified platform is Netapp.  They have pretty much always offered the standard range of protocols, including iSCSI at no extra cost.  However, even this platform isn’t a perfect solution; in previous years I’ve done performance and capacity comparisions of Netapp versus the competition for block storage and found it significantly lacking in horsepower, something I know they’ve worked on in recent years.

Should we care if the hardware isn’t fully integrated?  Does it matter that we’re putting components together like a 1970′s hi-fi system?  Well, like everything, it depends.  Building a hi-fi system out of individual components is pretty straightforward; the standards and links between each piece are well defined and consistent.  Pulling together NAS gateways and storage from different vendors is fraught with support issues and taking a single “unified” product guarantees support is from a “single throat to choke”, even if each piece isn’t best of breed.  It also means simplified management too.

Market Positioning

Unified storage products are great for smaller organisations, especially those without dedicated storage teams.  However where performance and scale are important, unified arrays are probably not the optimum solution.  As with everything in IT, there’s no right or wrong way, just many shades of grey.

What Primary De-Duplication is

De-duplication has typically been used to reduce the size of infrequently accessed archive data in dedicated archive appliances.  Blocks of identical data are “deduped” by removing them and only retaining pointers to a single physical copy on disk.  De-duplication appliances benefit from storing lots of identical (typically read-only) data such as that generated by backup appliances.  However de-duplication is also valuable in other environments were data can be duplicated, such as email archives and virtual environments.

How Compellent Could Do It

There are a number of features of the Compellent architecture that could enable de-duplication including:

• Snapshot Support – Storage Center already supports snapshots. This functionality creates point-in-time images of LUNs, only retaining pointers to shared blocks of data in the way de-duplication works.
• Metadata – The architecture already retains metadata on referencing LUNs, I/O activity and so on.  It wouldn’t be difficult to extend that to include a unique hash code per block.
• Write New – All changed blocks are written as new blocks of data.  Old data is simply invalidated unless it forms part of a snapshot.  Therefore, if a block of data is referenced by multiple LUNs and any LUN is updated, the changed data would be re-written as a new block and the old retained for other LUN references.  Over time the level of de-duplication would reduce.
• Background Processing – the existing storage controllers already run scheduled tasks to manage data progression, moving blocks between tiers of storage dictated by historical usage patterns.  It would be simple to add another task to scan and consolidate blocks with identical hash codes.

De-duplicating primary data has certain risks.  There is, for example, a risk of creating hot-spots of data access, where shared blocks become heavily accessed.  This can occur in Virtual Desktop implementations, for example.  Netapp introduced PAM (Performance Acceleration Module) cards to get over this kind of problem as their architecture isn’t capable of the granular i.e. block) level of data placement required to overcome this issue.  The Compellent architecture can do this already by promoting “hot blocks” to a faster tier of storage within a single LUN.  This ability is a key differentiator over other de-duplication implementations that would make Compellent hardware suitable for primary de-duplication.

Of course all this discussion is pure speculation as I have no prior knowledge of Compellent’s roadmap or futures strategy.  It is fun to try and second-guess things, though, and you never know, maybe this will become a feature in the future.

In some respects this move makes sense for EMC, if Netapp are seen as their major competitor in this market.  However as we know, performing this kind of code merge isn’t simple and Netapp themselves have had problems achieving code delivery dates (posts here, here and here).  Perhaps these problems are typical of hardware vendors that are looking to squeeze more out of their increasingly legacy looking hardware.

Of course the biggest confirmation to this theory comes from moves by HP.  They have purchased many new niche storage vendors over the last couple of years, culminating with 3Par only last month.  This is a clear indication that their storage portfolio needed strengthening (and in places replacing) and they’ve acquired new technology to achieve this.

I doubt sales for EMC and Netapp are about to drop off a cliff any time soon, but I would suggest that continuing to lever new features into old architectures is a long term recipe for disaster.  Eventually EMC & Netapp will need a new approach.  Before everyone leaps on my comments, I’ll just remind you all that EMC promised discussed FASTv2 at EMC World in 2009. It has yet to be released as GA code on Symmetrix.  That’s 18 months people.  The world has moved on and FAST is not the only block-level tiering product in town, making it an also-ran rather than an innovation.  Maybe that’s the issue with legacy hardware.

Please Note:  Due to the high level of Spam comments I am receiving (even with captcha-style technology enabled), all commenters will be required to register.  Hopefully I won’t have to force this restriction for too long.

Update: EMC PR have contacted me and asked me to clarify some errors in this post

I have been asked to clarify by EMC PR that FAST wasn’t announced in April 2009 at EMC World but was actually announced in December 2009 as part of this press release.  This is not entirely true.  In fact, FAST was mentioned in the April 2009 release on V-MAX, which can be found here:

http://www.emc.com/about/news/press/2009/20090414-01.htm

The EMC bloggers also commented about it (as did I).  Barry Burke’s post is probably the most detailed:

http://thestorageanarchist.typepad.com/weblog/2009/04/1059-fully-automated-storage-tiering-fast.html

My post is here:

http://www.thestoragearchitect.com/2009/04/14/enterprise-computing-emc-announced-next-generation-v-max-architecture/

So, irrespective of the “official” release date, EMC were selling the FAST story from April 2009 onwards.  As for FASTv2, take a look at this article from Gestalt IT, dated May 2009:

http://gestaltit.com/featured/top/gestalt/emc-symmetrix-vmax-fast-virtual/

FASTv2 was being discussed at this point.  This wasn’t an “official” EMC announcement, but FASTv2 was being discussed at EMC World 2009.

Finally, I’ve been asked to clarify the fact that FASTv2 was GA as from August 24th. This is the press release I’ve been directed to:

http://www.emc.com/about/news/press/2010/20100824-01.htm

It isn’t very clear what the new features of FAST are from this post, and strangely, neither Barry Burke or Chuck Hollis chose to blog about it.  I believe the release only covered the mid-range products, as this Register article discusses:

http://www.theregister.co.uk/2010/08/24/emc_fast_2_flare_30/

So, agreed, FASTv2 is available for CLARiiON & Celerra, but not for Symmetrix.

I’ve tweaked the wording in the post to reflect the specific mid-range reference.  I’ll leave it to your own judgement as to whether you’d rather go with the “official” release dates or the “unofficial” release dates.

I asked Chad the question – what happens if customers have heterogeneous environments?  The answer from Chad was “Good luck to them” – a comment I take to mean – you’re on your own.  Now Chad did and will point to the fact that there are vendors out there who provide 3rd party solutions equivalent to UIM in environments that are not pure VCE (VMware, Cisco, EMC), however is that a fair direction for EMC to take?

Personally, I feel that EMC are effectively telling customers that they adopt the Vblock model (and UIM), or they will be left out in the cold.  It’s an example of further moves to lock end users into their proprietary stack of technology.  However although Chad intimates that alternatives to UIM exist, unfortunately they will never be as good as UIM for one simple reason:

VMware, Cisco and EMC own and understand the APIs used to drive their products.  They can (and I suspect probably do) retain certain documentation that enables them to have a competitive advantage over 3rd party vendors attempting to add equivalent functionality.  In addition, they can also make microcode and software changes to those APIs to provide any feature that might be needed to make UIM a better product than their competitors.  This ability always gives them the edge over any other vendor looking to offer similar functionality to UIM.

Of course, why shouldn’t EMC/VMware & Cisco do this?  After all, they have the right to do anything with their products that they choose and far be it from me to tell any vendor what their product & marketing strategies should be.  But it is worth looking back to the attempts by many companies to offer unified storage management tools; no hardware or software vendor has yet provided a single unified storage management tool that works across heterogeneous environments.  Customers still retain multiple products for storage provisioning, in many cases continuing to make use of CLIs rather than graphical interfaces.  So, the chances of there being a successful generic virtualisation orchestration product out there are slim to none.

So what is the solution?  For EMC, their answer would be to take Vblock and UIM.  But what happens if you take that route and build all your processes around UIM, then decide to change your storage vendor?  At that point you have nowhere to go other than to re-engineer your entire orchestration process, creating a very high cost and barrier to change.

Here’s one other point to ponder; in February, EMC sold off a number of the Ionix tools to VMware.  So if these products were intended to be part of vCenter, why is UIM coming from EMC?  There’s one easy answer to this; if VMware sold the product, their other partners (like Netapp) could push for integration of their products into UIM.  By selling UIM as an EMC tool, this support will never happen and so Netapp are forced to either write their own tools or partner with 3rd party vendors.  In this way the VCE coalition can continue to control their market.

So do you think vendor lock-in is a good thing?

This may seem like a bold statement to make, however we need to look forward to where the industry is headed.  First of all, vendors want us to buy their unified hardware stacks; it represents that move back to a consolidated architecture that kept one vendor dominant in the mainframe days – IBM.  “No-one gets fired for buying IBM” the saying goes (or used to go), demonstrating how IBM was seen as the data centre supplier for all things computing in the 70′s and 80′s.  Of course we know that politics within organisations and the cost of IBM hardware eventually broke the monopoly, but the status quo worked well for many companies for many years.

Now, Cisco, EMC, VMware, HP, Oracle and potentially many others want to own your data centre.  They want you bought into their computing stack.  Over time, I suspect many of those same companies want to move you to their cloud infrastructure offerings, even if they don’t offer them today.  This will be both directly and indirectly.  There will be the direct model, where the vendor offers cloud services to you under their name; there will be the indirect model where their technology powers the cloud provider, or is offered as a service.  It’s at this point the 3Par acquisition becomes much more interesting.

3Par already have many customers in the cloud services sector.  In fact they sell their hardware on the virtues of multi-tenancy, reduced cost through thin provisioning, tight integration with virtual hypervisors and so on.  In this growth sector of the industry, cost is a key driver and no end user or company will pay more than they need for storage.  This means Enterprise arrays like those from Hitachi and EMC won’t play a central role in this future, but rather storage devices which provide the highest efficiency will.  Where do all the major players stand?

• EMC have entered the market with a brand new platform – Atmos.  Although withdrawn as a direct service, Atmos continues to be available from partners.  EMC have chosen to use their own technology as the foundation for cloud.  In addition, VPLEX provides the ability to virtualise the storage layer, including federation features that fit well with VMware.
• HP have a strong blade server offering for their cloud infrastructure.  Matrix provides orchestration for the server, network and some parts of the storage layer, however this work is incomplete and doesn’t fit well with the high end XP arrays.  Slotting 3Par into the storage layer would provide a storage platform well suited to HP Converged  Infrastructure.  It means EVA can be quietly dropped and XP can be retained (in whatever future guise) for high end customers (including mainframe) and if required, gradually dropped.
• Cisco have chosen to partner with EMC rather than acquire storage technology itself.  In fact, looking and both EMC and Cisco, they need each other; EMC have no server platform, Cisco have no storage; it’s a mutually beneficial arrangement, a bit like Jack Sprat and his wife.  At this stage, Cisco could have purchased 3Par and provided and end-to-end solution, but clearly that would be a big step and would require kicking EMC to the kerb, something they obviously don’t want to do (yet).
• Hitachi have server and storage offerings, however Blade Symphony is mainly sold in domestic Japan and not widely advertised globally.  They do have the potential to provide an end-to-end offering as Hitachi also sell networking equipment.  Key for Hitachi will be credibility in a market they don’t currently play in.
• IBM should have all the components of a consolidated infrastructure but there doesn’t appear to be a lot of discussion about their offerings.  They appear to have two strategies – Dynamic Infrastructure and Cloud Computing but their offerings aren’t clear.
• Dell clearly wanted 3Par to fit into their medium to high-end storage offerings.  Today Equallogic has successfully met their SMB requirements, but they OEM technology from EMC (CLARiiON and Symmetrix) for the rest.  Acquiring 3Par would remove that dependency and allow Dell to offer end-to-end technology as their own products.
• Netapp have a self-proclaimed unified architecture that does fit well with virtualisation from VMware.  However they don’t own any other parts of the technology stack and so must partner to deliver unified offerings.  Netapp are covering all bases by offering solutions with VMware, Microsoft and Citrix, but none of these could be described as the unified stack other vendors have.
• Oracle already provide an integrated infrastructure based around their Exadata acquisitions and of course all of the Sun Microsystems technology, however I’m not sure many companies would see the Oracle offerings as other than tied directly to their database platform and not for virtualisation.

Overall, 3Par fit the requirements of HP & Dell to provide integrated technology offerings.  The move to the cloud will require leaner and efficient storage products, plus tight integration and orchestration.  It’s all about positioning today for bigger returns tomorrow.

• Monolithic – this architecture is characterised by Hitachi USP, HP XP & EMC DMX and consists of a shared memory architecture and multiple redundant components.
• Multi-Node – these devices use loosely coupled storage “nodes” with a high-speed interconnect providing scalability by adding extra nodes to the storage “cluster”.  Products in this category include EMC VMAX and 3Par InServ.
• Closely Coupled Dual Controller – this is the typical “modular” storage architecture characterised by IBM DS8000, EMC CLARiiON, Hitachi AMS and HP EVA.
• Loosely Coupled Dual Controller - this category describes technology that are capable of device failover but aren’t closely coupled to enable individual LUN failover as the Closely Coupled model permits.  This category is characterised by arrays such as Netapp FAS filers and Compellent Storage Center.
• Single Controller – this category covers devices that act as standalone products, including SOHO storage devices such as the Iomega IX4 & Data Robotics Drobo series.

The above list isn’t exhaustive and it’s my own personal categorisation.  There are many more vendors of technology than I’ve listed here.  In addition, none of these lists qualify as “Enterprise” in their own right.  The use of this term is a hotly debated subject.

Monolithic Architectures

EMC DMX High Level Architecture

Monolithic arrays use a shared cache architecture to connect front-end storage ports to back-end disk.  This is shown clearly in the architecture diagrams shown here, representing the internal connectivity of the EMC DMX  and Hitachi USP storage arrays.  Each of the memory units is connected to each of the front-end directors and the back-end disk directors.  Hitachi divide their cache into two halves for Clusters 1 & 2 in the array; EMC have up to eight cache modules.  This architecture has positive and negative benefits; firstly having director connections connecting to all cache modules ensures resources aren’t fragmented;  unless cache becomes completely exhausted there’s always connectivity to another cache module to process a user request.  It also doesn’t matter on which port that request comes in; the cache module can process any request from any port to any back-end disk.  This connectivity is also beneficial in terms of failure.  If a cache module fails, for example, only the cache on that module is lost; in a fully deployed architecture the total cache would drop (by 1/8th in EMC’s case), but front and back-end connectivity would remain the same.  With this model it is possible pair up storage ports and have a single LUN presented from 1 or more ports with no performance impact; the path length between a storage port and disk adaptor will always be the same.

This any-to-any model also has disadvantages.  The connectivity is complex and therefore becomes expensive and requires overhead to manage and control the interaction between the various components.  In addition, there’s a limit to the practical scalability of this architecture.  With eight FE, BE and cache modules, there are 128 connections in place; (8x8x2).  Adding a single cache module requires an additional 16 connections; similarly, adding more front or back-end directors requires more connectivity.  Also monolithic arrays are based on custom components and custom design, increasing the ongoing maintenance and development costs for the hardware.

One other point to remember; front and back-end directors have their own processors.  It is possible for the traffic across the directors to be unbalanced and for some processors to be more heavily utilised than others.  I’ve seen a number of configurations where USP V FED ports are running at 100% processor utilisation due to to small block sizes.  This means manual load balancing is required both in initial host placement and subsequently as traffic load increases.  This fact is worth bearing in mind as we move to more highly virtualised environments as it is likely host port utilisation will start low and rise over time as more virtual machines are created.

Hitachi USP High Level Architecture

Now that the DMX platform has been put out to pasture in place of VMAX, it appears Hitachi are the only vendor continuing down the monolithic route.  Next time I’ll discuss Multi-Node arrays and why they may (or may not) be a replacement for today’s monolithic devices.

Data ONTAP History

At the point of reaching DOT8, Data ONTAP has been re-written to run off FreeBSD as the original GX code did.  This is a departure from the original Berkeley Net/2 code as documented in this post from Dave Hitz.  I have no idea how much of this version of the code was a re-write, but presumably porting over WAFL with all the bells and whistles it now has wasn’t an easy task.  This may go towards explaining why the current release of ONTAP took so long to come out.

Although the diagram shows the code base as being a single product, it isn’t.  There are still two modes, 7-mode, emulating  7G and cluster-mode emulating the GX product line.  These modes are non-interchangeable; you choose the one you want to use at system installation time and it’s fixed; no chance to change in the future.  As the IBM document explains (quote) “If a customer decides to change from one mode to another, the change is a transition rather than an upgrade (or downgrade).  Dual boot capabilities are not present, so the transition requires total reconfiguration of the storage system.  This can include backup and restore of user data”.

I think it would have been fairer to draw two parallel lines here as it appears there are still to pretty separate versions of code masquerading as a single marketing version.  So, the remainder of this discussion focuses on 7-mode.

Upgradeability

What happens if you want to take an existing system and upgrade it?  Well, depending on your hardware, you may or may not be able to perform an upgrade.  Systems such as the 6xxx models, 3×70 & 3×40 models are upgradeable, devices such as the 2050 are not.  There are also restrictions on the disk shelves that can be used too.  Should you choose to upgrade from your current 7G release, you can only move to 7-mode or build a new 8.0 installation, presumably on new hardware as you wouldn’t want to trash your existing environment.  Be aware though, that upgrade actually removes certain features.  For instance, SMB 2.0, IPv6 & IPSec are not supported.  They will reappear in a future release.  Does this mean writing these features in the ported version of WAFL was too hard or was taking too long?  Why else would you remove features from an upgrade only to replace them later?  One final upgradeability gotcha – Performance Acceleration Modules (PAM) are not supported with the initial version of 8.0.

Aggregates

As mentioned in my previous post, aggregates move to 100TB in size.  However there are many restrictions.

• Volume SnapMirror will not replicate between unlike aggregate types; so you can’t replicated to/from 32-bit to 64-bit aggregates.
• aggr copy and vol copy commands will not work between different formats.
• Flexvol size for volumes using de-duplication in 64-bit aggregates is limited to 16TB
• System root volumes can only reside on 32-bit aggregates.

On the positive side, Qtree SnapMirror and SnapVault do work between aggregate formats.

Aggregate Migration

Here’s IBM’s statement on migration of data: “Currently there is no direct migration path or conversion from 32-bit to 64-bit aggregates.  The following options can be used to migrate the data: qtree SnapMirror, SnapVault, ndmpcopy”. Each of these options also has limitations, which I don’t have time to go into, but you can read in the referenced document.

Benefits

I’ve been trying to find any benefits of upgrading to DOT8 from the customer’s perspective. For new installations, the increased aggregate size is obviously a benefit, but does come with restrictions.  There are now interface groups rather than VIFs and it appears snapshots can now be named.  Excluding these, I can’t see that DOT8 is anything more than a positioning exercise as Netapp continue to get the real features they wanted in this version into future releases.  This has been hinted at by other commentators.

Whilst I can see the benefits to Netapp of this move, I fail to see the benefit to the customer, who will have to suffer major migration headaches to realise what are small improvements from a major version upgrade.  I suspect many customers will chose to wait for 8.0.1, 8.1 or whatever version actually integrates the real improvements.  During that time, it offers more opportunities for the competition to be snapping closer to Netapp’s heels.

So, as part of their rebuttal, Netapp employees pointed me to two features; snapmirror migrate and Data Motion.  In the spirit of fairness, I have researched these two features using the only tools available to me – Netapp documentation – and here’s what I found.

SnapMirror Migrate

This feature is similar in nature to the concept of “failover” on large EMC arrays where a remote array takes over operation of a local LUN.  As the operation is taking place on NAS, things are slightly different.  Here’s a direct quote from the Data ONTAP System Command Reference:

“The first thing migrate will do is check the source and destination sides for readiness. Then, it will stop NFS and CIFS service to the source. This will prevent changes to the source volume’s data, which will make it appear to clients as though nothing has changed during the migration. It will run a regular SnapMirror transfer between the two volumes. At the end of the transfer, it will migrate the NFS filehandles, bring the source offline, and make the destination volume writable.”

So what does this mean?  Presumably the first thing that happens is that all I/O operations are suspended and the hosts are unable to write to their target volumes.  This introduces a whole range of concerns about data integrity and data loss if any kind of outage occurs while the copy is taking place.  It isn’t clear whether the volume effectively becomes unavailable.  Here’s the next quote:

“The migrate process will not take care of renaming or exporting the destination volume.
As a result of this process, the source volume home will be taken offline, and NFS service to this filer will be stopped during the transfer. CIFS service on the source volume will be terminated and CIFS will have to be set up on the destination.”

Clearly not a transparent, non-distruptive migration then.  I’d suggest that SnapMirror Migrate is purely an extension the ability to move data to other filers and not an “in-filer” migration function due to the impacts on file handles and I/O.  Imagine moving 16TB volumes around using this approach; no thank you.

Data Motion

There’s not much to say about this product – and this time I quote from the Data ONTAP 8.0 7-mode release notes:

“Data Motion is not supported

Data Motion was introduced in a previous release, but it is not supported in Data ONTAP 8.0 7-Mode.
The Data Motion feature integrates virtual storage, mirroring, and provisioning software technologies so that you can perform data migrations non-disruptively in both physical and virtual environments.”

Whoops!  Guys, don’t try and sell me on a technology you’ve since discontinued!  I’d be pretty miffed if I’d built my process around this tool and was then told I could no longer use it.  The release notes imply Data Motion will be back again as a feature in future releases.  It really demonstrates that the ONTAP code is struggling when features have to be removed and re-introduced over time.

I stick by my assertion that Data ONTAP is becoming unwieldy and I’m obviously not the only one of this opinion.  Netapp need to pull some serious rabbits out of the colloquial hat otherwise they seriously risk losing ground to better competition.