Hitachi Non-Disruptive Migration

It’s probably fair to say that data migration is one of the most tedious tasks in storage management.  Moving data from one array to another, simply to decommission a piece of hardware or for load and capacity balancing, doesn’t ever inspire me. But it’s a necessary evil and one I’ve done many times.  Unless the existing storage configuration has been well planned, migrations can take months to accomplish, at high cost, in order to avoid outages and maintain data integrity.

So during last year’s Hitachi Influencers event in San Jose, I was extremely interested to see that the migration issue might have finally been put to bed by one of the vendors.  This week we saw the release of last year’s demo; Hitachi’s non-disruptive migration service.

At it’s heart the concept isn’t that complicated; by enabling a new target storage array to act as if it is another set of data paths through the SAN to the original array, a LUN can be mirrored away from the source, onto the target, while the data is mirrored in the background.  Once the mirroring is complete, drop the first set of paths that connect to the LUN and in effect the LUN “migrates” to the new target array in a completely transparent fashion.

This migration process works because Hitachi are able to both virtualise a LUN through another storage array (technology that has existed for quite some time) and move the World Wide Name (WWN) of the source array across and present it out from the target.  In essence, the target array “spoofs” the host into believing that the WWN of the old array still exists on the network.

What’s interesting here is that the migration process is making use of standard Fibre Channel protocols and so the migration process could be used to move data from any array, not just Hitachi products.  There’s also no disruption to the existing configuration, as the virtualisation of the source array can be achieved behind the scenes without impacting the existing configuration.  There are of course some restrictions or issues to consider; migrating LUNs that are array replicated needs to be thought through and of course the target array continues to appear with a different WWN, which could be confusing during and after migration work.

However, the power of this technology is the ability to avoid cost.  Hitachi’s storage economist, Dave Merrill, estimates that storage costs around $7K-15K per TB to migrate between arrays, which is way more than new storage costs to acquire in the first place.  Consider that many customers will put a new vendor on the hook to cover migration, then this service puts Hitachi in a much stronger position when tendering for swap-out business.

As with everything, what’s usually more interesting is not what can be achieved today, but what can be done tomorrow.  The ability for an Hitachi array to offer out virtual WWN port names means that an entire physical array could be split into multiple virtual arrays, in a similar way to the Multistore feature offered by Netapp.  Now, instead of having WWNs that match to a physical device, a virtual array could be created.  This virtual array can be managed with its own QOS, or migrated or shared between hardware platforms, without the user having to have any knowledge of where the data is actually sitting.  Imagine using a single physical array to create secure multi-tenant virtual arrays with the ability to manage each with an individual QOS (something that Multistore can’t do today).

Although I can’t claim to have a crystal ball, I did predict this kind of feature on the release of the VSP in 2010 (see my previous article where I discuss this possibility).  I have no idea whether Hitachi will deliver this feature, but I hope they do.

My only disappointment with Non-Disruptive Migrations is that it may have come to late for many organisations.  As virtualisation becomes more prevalent, migrations will be achieved using (for example) Storage vMotion in the hypervisor, negating the need to care whether the array can perform the migration on the host’s behalf.  However server virtualisation isn’t everywhere and the option of multi-tenancy is still a powerful one, even with the ability to virtualise the server.

Related Posts

• It’s About Time (and Money)
• No More Migration Tears

Disclaimer: Last year I attended Hitachi’s Influencer Forum in San Jose.  Hitachi paid for my travel and accommodation as well as most meals.  Most of the content of this event was NDA only and so hasn’t been discussed until now.  There is no requirement on me to blog about any of the content presented during the event.  I am not otherwise employed by Hitachi, or compensated for my time.

Here’s the URL again:

http://blogs.hds.com/claus/2011/09/putting-hdd-product-trends-into-perspective-a-subsystem-view.html

HDS/Hitachi I believe were an investor in BlueArc, now they’ve acquired them.  I think this is a good move for both companies for a number of reasons.  Firstly, it marks the continuing change in HDS’ strategy, which now includes acquiring companies, albeit companies that they have an existing relationship with.  This approach may seem cautious but perhaps it’s a good thing to take time to understand the people, technology and culture, although HDS could have made their move earlier.  Second, I believe BlueArc suffers in a market where (large enterprise) customers expect longevity.  Committing to a new platform that suddenly has little or no development could be a costly mistake.  There’s also the issue of global support and maintenance to consider too.  Being part of Hitachi will give customers more reassurance.

Some time ago, I put the following diagram together as part of work for a customer.  it was meant to show how the Hitachi technology can operate on a number of layers, from the physical hardware upwards.  Although I have no specific inside knowledge, I expect Hitachi will converge the physical layer into a single device at some stage.  Chris Mellor’s article hinted at this (although I expect this slide is no longer accurate) and developments in their technology imply this too.  The AMS2XXX range certainly became more USP-like in its last release.

Operating a single hardware platform makes sense as HDS can abstract the hardware from the logical view using tools like HTSM and UVM.  Presentation is then through native FC/FCoE or using gateway products to provide NAS and object connectivity.  VSP represents a much more scalable architecture than the USP was, one of the key features being the abstraction of the front and back-end processors from the physical interfaces.  This scalability easily supports multi-protocol shared environments.

Other vendors have gone down the route of doing scale-out through multiple interconnected nodes.  I think this is a good approach and it could be argued that Hitachi have taken another way and gone for single monolithic arrays.  I don’t believe this is the case as the VSP is already a cluster of two nodes and from the PCIe backplane architecture can presumably scale to more nodes as required.  They are however more closely coupled than other implementations.  If Hitachi delivers on the potential promise of HAM, then clustering/nodes can be implemented and the single monolithic array isn’t an issue.  Now that Hitachi own BlueArc, they can integrate the functionality to ensure any NAS presentation can be mapped back to any storage array, with seamless migration.

So in summary, the future looks good, I’m looking forward to seeing how Hitachi capitalise on their new investment.

Disclaimer: I’ve been engaged with HDS as a blogger, including trips to Japan and the US.  I have worked for HDS UK delivering consultancy to their clients.

The event itself was well organised and informative.  Hitachi weren’t afraid to be put on the spot and answer those hard questions bloggers like to ask.  This open approach is refreshing and unfortunately so few vendors are prepared to do it.

Anyway, back to the subject in hand.  During the course of the two days, the content was deep and varied, covering block hardware (VSP), file and content (HCP and HDI – more on that later), healthcare, servers and Storage Economics.  Rather than lump everything into a single post, I’ll start around the discussion of VAAI support.

Everybody’s Doing It

Yes, it’s true to say anyone who’s anyone is supporting VAAI.  Check out Stephen Foskett’s great post that covers VAAI support across the major vendors.  Clearly Hitachi can’t afford *not* to be supporting VAAI as it is essential for large-scale virtualised environments.  The reason for this is the need to move data around both within and between hypervisors to support load balancing and cloning.  In addition, VAAI support increases the granularity of file locking from the LUN (i.e. a complete VMFS object) to the block level with two benefits; it reduces contention on the VMFS (so improving performance) and also allows datastores to be scaled to more than 2GB in size (although there are other restrictions preventing this).  Lastly VAAI reduces the need to write “empty” data by offloading block writes of zeros to the array.  In those arrays that support thin provisioning this should also mean reduced storage consumption as those “empty” blocks of data will simply be marked as logically in use.

So why is Hitachi so keen to talk about VAAI?  Well there are a number of reasons.  Most obviously they want all their customers out there to know they “get” virtualisation and VMware – definitely a marketing position.  It also means being able to use the heritage Hitachi have in working with mainframe environments, where extent-level locking was the norm.  Possibly more important in my mind is the fact that VAAI will be essential to driving the scalability and the performance out of the new VSP.

3D Scaling

Following the VSP release last year, I wrote this post talking about some of the internal VSP architecture that allows it to scale better than previous models, including support for external storage (the third dimension).  Extracting every ounce of performance from an array is something customers will surely want, but if the server platform can’t produce that demand then the VSP becomes an expensive commodity.  By supporting VAAI, Hitachi are ensuring that virtualised environments can offload as much workload as possible to the array, thereby fully utilising the performance capabilities of the storage hardware.

Without a doubt, high performance is a Good Thing, but let’s face it you can have too much of a good thing and this is the area where I have my concerns.  A VSP can scale to support multiple virtual environments; it would therefore be possible for virtualisation administrators to overload the storage array by instancing multiple cloning and vMotion acitivities at the same time.  What controls are there to prevent this?  It would appear there are none, or at least The Heff wasn’t sure what they were.

Storage I/O Control

What we need is a control mechanism for the VAAI features.  When all I/O came from outside of the array, things were simple; too much I/O could be throttled by restricting queue depth per host.  That’s not to say an array couldn’t be overwhelmed, of course they could, especially if remote replication was used.

Having been involved on some in-depth discussions in the past with the engineers in Hitachi Japan, I *know* there are throttling mechanisms in other parts of the VSP design.  Workload is prioritised to ensure that higher priority activities (like responding to cache destages for replication) are handled before other work.  Therefore I’d expect that the VAAI functions would be treated as lower priority and handled after normal I/O activity.  Heff has promised to go away and confirm this, so hopefully we’ll see the answer soon.

One other thought – the subject of security.  The VAAI copy features are implemented through the SCSI EXTENDED COPY command.  From what I have read (and posted here) there are no security controls to prevent any user from issuing this SCSI copy command.  So, there’s nothing to stop me writing a SCSI call to copy data from another LUN on the array into a LUN I have access to.  I would hope I am wrong on this assumption and that there would be a higher level check within Fibre Channel/iSCSI/FCoE to validate I have access to the source and target LUNs.  If there is, unfortunately I haven’t been able to find it.  The upshot of this (potential) loophole is that I could obtain access to secure data.  Worse, I could change or overwrite that data either destructively or for my benefit.  Hopefully someone will prove me wrong.

So VAAI is a good thing; Hitachi demonstrated VAAI working on a vMotion and datastore creation task.  What we need to see next is a demonstration at scale.

For reference you can find a copy of Heff’s presentation here.

Disclosure:  The Hitachi Bloggers’ Day was an invitation-only event.  My accomodation and meals (but not my travel) were paid for by Hitachi, however I was not compensated for my time.

Click here to view the embedded video.

As you can imagine, with the release of a new product, there’s a lot to take in. Fortunately I’ve been privileged to be provided with information from both HDS and HP on the hardware prior to release.  Regardless of which vendor you take the hardware from, some of the highlights include:

• 2.5″ drive support
• Redesigned core processor architecture
• Overall 2x improved performance
• Reduced Power consumption
• Higher scalability
• Block level-tiering

If you want to get the low-down on the hardware itself, have a look at Nigel Poulton’s great post here.

What doesn’t seem immediately obvious is the difference in architecture between this and the predecessor USP V.  With everyone’s focus on speeds and feeds, this fundamental change can easily be overlooked.  I recommend you look at my first two posts on array architectures:

• Choosing Between Monolithic and Modular Architectures Part I
• Choosing Between Monolithic and Modular Architectures Part II

Part II will prove to be more useful for those people who are bound to want to compare the VSP/9500 with VMAX from EMC.  Rather than rush into a post today, I’ll be posting on the details over the next 48 hours, as there are software changes that also are significant in this release.  I’m also hoping to have some interviews in place, so look out for those too.

This may seem like a bold statement to make, however we need to look forward to where the industry is headed.  First of all, vendors want us to buy their unified hardware stacks; it represents that move back to a consolidated architecture that kept one vendor dominant in the mainframe days – IBM.  “No-one gets fired for buying IBM” the saying goes (or used to go), demonstrating how IBM was seen as the data centre supplier for all things computing in the 70′s and 80′s.  Of course we know that politics within organisations and the cost of IBM hardware eventually broke the monopoly, but the status quo worked well for many companies for many years.

Now, Cisco, EMC, VMware, HP, Oracle and potentially many others want to own your data centre.  They want you bought into their computing stack.  Over time, I suspect many of those same companies want to move you to their cloud infrastructure offerings, even if they don’t offer them today.  This will be both directly and indirectly.  There will be the direct model, where the vendor offers cloud services to you under their name; there will be the indirect model where their technology powers the cloud provider, or is offered as a service.  It’s at this point the 3Par acquisition becomes much more interesting.

3Par already have many customers in the cloud services sector.  In fact they sell their hardware on the virtues of multi-tenancy, reduced cost through thin provisioning, tight integration with virtual hypervisors and so on.  In this growth sector of the industry, cost is a key driver and no end user or company will pay more than they need for storage.  This means Enterprise arrays like those from Hitachi and EMC won’t play a central role in this future, but rather storage devices which provide the highest efficiency will.  Where do all the major players stand?

• EMC have entered the market with a brand new platform – Atmos.  Although withdrawn as a direct service, Atmos continues to be available from partners.  EMC have chosen to use their own technology as the foundation for cloud.  In addition, VPLEX provides the ability to virtualise the storage layer, including federation features that fit well with VMware.
• HP have a strong blade server offering for their cloud infrastructure.  Matrix provides orchestration for the server, network and some parts of the storage layer, however this work is incomplete and doesn’t fit well with the high end XP arrays.  Slotting 3Par into the storage layer would provide a storage platform well suited to HP Converged  Infrastructure.  It means EVA can be quietly dropped and XP can be retained (in whatever future guise) for high end customers (including mainframe) and if required, gradually dropped.
• Cisco have chosen to partner with EMC rather than acquire storage technology itself.  In fact, looking and both EMC and Cisco, they need each other; EMC have no server platform, Cisco have no storage; it’s a mutually beneficial arrangement, a bit like Jack Sprat and his wife.  At this stage, Cisco could have purchased 3Par and provided and end-to-end solution, but clearly that would be a big step and would require kicking EMC to the kerb, something they obviously don’t want to do (yet).
• Hitachi have server and storage offerings, however Blade Symphony is mainly sold in domestic Japan and not widely advertised globally.  They do have the potential to provide an end-to-end offering as Hitachi also sell networking equipment.  Key for Hitachi will be credibility in a market they don’t currently play in.
• IBM should have all the components of a consolidated infrastructure but there doesn’t appear to be a lot of discussion about their offerings.  They appear to have two strategies – Dynamic Infrastructure and Cloud Computing but their offerings aren’t clear.
• Dell clearly wanted 3Par to fit into their medium to high-end storage offerings.  Today Equallogic has successfully met their SMB requirements, but they OEM technology from EMC (CLARiiON and Symmetrix) for the rest.  Acquiring 3Par would remove that dependency and allow Dell to offer end-to-end technology as their own products.
• Netapp have a self-proclaimed unified architecture that does fit well with virtualisation from VMware.  However they don’t own any other parts of the technology stack and so must partner to deliver unified offerings.  Netapp are covering all bases by offering solutions with VMware, Microsoft and Citrix, but none of these could be described as the unified stack other vendors have.
• Oracle already provide an integrated infrastructure based around their Exadata acquisitions and of course all of the Sun Microsystems technology, however I’m not sure many companies would see the Oracle offerings as other than tied directly to their database platform and not for virtualisation.

Overall, 3Par fit the requirements of HP & Dell to provide integrated technology offerings.  The move to the cloud will require leaner and efficient storage products, plus tight integration and orchestration.  It’s all about positioning today for bigger returns tomorrow.

• Monolithic – this architecture is characterised by Hitachi USP, HP XP & EMC DMX and consists of a shared memory architecture and multiple redundant components.
• Multi-Node – these devices use loosely coupled storage “nodes” with a high-speed interconnect providing scalability by adding extra nodes to the storage “cluster”.  Products in this category include EMC VMAX and 3Par InServ.
• Closely Coupled Dual Controller – this is the typical “modular” storage architecture characterised by IBM DS8000, EMC CLARiiON, Hitachi AMS and HP EVA.
• Loosely Coupled Dual Controller - this category describes technology that are capable of device failover but aren’t closely coupled to enable individual LUN failover as the Closely Coupled model permits.  This category is characterised by arrays such as Netapp FAS filers and Compellent Storage Center.
• Single Controller – this category covers devices that act as standalone products, including SOHO storage devices such as the Iomega IX4 & Data Robotics Drobo series.

The above list isn’t exhaustive and it’s my own personal categorisation.  There are many more vendors of technology than I’ve listed here.  In addition, none of these lists qualify as “Enterprise” in their own right.  The use of this term is a hotly debated subject.

Monolithic Architectures

EMC DMX High Level Architecture

Monolithic arrays use a shared cache architecture to connect front-end storage ports to back-end disk.  This is shown clearly in the architecture diagrams shown here, representing the internal connectivity of the EMC DMX  and Hitachi USP storage arrays.  Each of the memory units is connected to each of the front-end directors and the back-end disk directors.  Hitachi divide their cache into two halves for Clusters 1 & 2 in the array; EMC have up to eight cache modules.  This architecture has positive and negative benefits; firstly having director connections connecting to all cache modules ensures resources aren’t fragmented;  unless cache becomes completely exhausted there’s always connectivity to another cache module to process a user request.  It also doesn’t matter on which port that request comes in; the cache module can process any request from any port to any back-end disk.  This connectivity is also beneficial in terms of failure.  If a cache module fails, for example, only the cache on that module is lost; in a fully deployed architecture the total cache would drop (by 1/8th in EMC’s case), but front and back-end connectivity would remain the same.  With this model it is possible pair up storage ports and have a single LUN presented from 1 or more ports with no performance impact; the path length between a storage port and disk adaptor will always be the same.

This any-to-any model also has disadvantages.  The connectivity is complex and therefore becomes expensive and requires overhead to manage and control the interaction between the various components.  In addition, there’s a limit to the practical scalability of this architecture.  With eight FE, BE and cache modules, there are 128 connections in place; (8x8x2).  Adding a single cache module requires an additional 16 connections; similarly, adding more front or back-end directors requires more connectivity.  Also monolithic arrays are based on custom components and custom design, increasing the ongoing maintenance and development costs for the hardware.

One other point to remember; front and back-end directors have their own processors.  It is possible for the traffic across the directors to be unbalanced and for some processors to be more heavily utilised than others.  I’ve seen a number of configurations where USP V FED ports are running at 100% processor utilisation due to to small block sizes.  This means manual load balancing is required both in initial host placement and subsequently as traffic load increases.  This fact is worth bearing in mind as we move to more highly virtualised environments as it is likely host port utilisation will start low and rise over time as more virtual machines are created.

Hitachi USP High Level Architecture

Now that the DMX platform has been put out to pasture in place of VMAX, it appears Hitachi are the only vendor continuing down the monolithic route.  Next time I’ll discuss Multi-Node arrays and why they may (or may not) be a replacement for today’s monolithic devices.

I wasn’t really sure what to expect from this event, as the website (which is Japanese only) isn’t too helpful.  Google Translate does a reasonable job on most of the content but it doesn’t really give you a clue about the event itself and the benefits from attending.

Hiroaki Nakanishi Keynote Speech

The event started with a keynote speech from Hiroaki Nakanishi, Hitachi President and Representative Executive Officer in a huge auditorium holding thousands of people.  I suspect there were few western delegates there, however we were provided with translation equipment as the presentation was in Japanese.  The historical and forward looking speech detailed Hitachi’s history, highlighting some interesting products and clearly demonstrating how Hitachi technology is pervasive in Japanese life.  What was most surprising (and probably more surprising for those people who don’t believe Hitachi Data Systems (HDS) forms an integral part of the company) is that the Hitachi products (including Hitachi Content Platform) received a significant amount of airtime.

The convention also showcases the technology products produced by Hitachi and along with the other attendees we were taken on a tour of the showfloor.  Products demonstrated included healthcare (see the attached pictures), transportation (rail rollingstock and automotive components), robotics (again the picture), power generation and a wide range of Information Technology products.

Within the Japanese domestic market, Hitachi already sells blade servers and networking equipment.  I was able to spend time and look at the blade systems available, including a high-density 40-server blade system (pictured).  These devices (and potentially the networking equipment) will form part of the Hitachi Unified Computing Platform, soon to be released globally.

The uValue convention was a pleasant surprise; I hadn’t understood the scale and diversity of the products Hitachi produced.  I await with interest the release of UCP as there is already a maturity in the Hitachi offering.   As usual the devil will be in the detail.

In the meantime, enjoy this short video of me, other attendees and Hitachi/HDS staff, enjoying our food over the 4 days, plus a Flickr link to pictures I took.

Hitachi uValue 2010 Pictures

Click here to view the embedded video.