Hitachi Non-Disruptive Migration

It’s probably fair to say that data migration is one of the most tedious tasks in storage management.  Moving data from one array to another, simply to decommission a piece of hardware or for load and capacity balancing, doesn’t ever inspire me. But it’s a necessary evil and one I’ve done many times.  Unless the existing storage configuration has been well planned, migrations can take months to accomplish, at high cost, in order to avoid outages and maintain data integrity.

So during last year’s Hitachi Influencers event in San Jose, I was extremely interested to see that the migration issue might have finally been put to bed by one of the vendors.  This week we saw the release of last year’s demo; Hitachi’s non-disruptive migration service.

At it’s heart the concept isn’t that complicated; by enabling a new target storage array to act as if it is another set of data paths through the SAN to the original array, a LUN can be mirrored away from the source, onto the target, while the data is mirrored in the background.  Once the mirroring is complete, drop the first set of paths that connect to the LUN and in effect the LUN “migrates” to the new target array in a completely transparent fashion.

This migration process works because Hitachi are able to both virtualise a LUN through another storage array (technology that has existed for quite some time) and move the World Wide Name (WWN) of the source array across and present it out from the target.  In essence, the target array “spoofs” the host into believing that the WWN of the old array still exists on the network.

What’s interesting here is that the migration process is making use of standard Fibre Channel protocols and so the migration process could be used to move data from any array, not just Hitachi products.  There’s also no disruption to the existing configuration, as the virtualisation of the source array can be achieved behind the scenes without impacting the existing configuration.  There are of course some restrictions or issues to consider; migrating LUNs that are array replicated needs to be thought through and of course the target array continues to appear with a different WWN, which could be confusing during and after migration work.

However, the power of this technology is the ability to avoid cost.  Hitachi’s storage economist, Dave Merrill, estimates that storage costs around $7K-15K per TB to migrate between arrays, which is way more than new storage costs to acquire in the first place.  Consider that many customers will put a new vendor on the hook to cover migration, then this service puts Hitachi in a much stronger position when tendering for swap-out business.

As with everything, what’s usually more interesting is not what can be achieved today, but what can be done tomorrow.  The ability for an Hitachi array to offer out virtual WWN port names means that an entire physical array could be split into multiple virtual arrays, in a similar way to the Multistore feature offered by Netapp.  Now, instead of having WWNs that match to a physical device, a virtual array could be created.  This virtual array can be managed with its own QOS, or migrated or shared between hardware platforms, without the user having to have any knowledge of where the data is actually sitting.  Imagine using a single physical array to create secure multi-tenant virtual arrays with the ability to manage each with an individual QOS (something that Multistore can’t do today).

Although I can’t claim to have a crystal ball, I did predict this kind of feature on the release of the VSP in 2010 (see my previous article where I discuss this possibility).  I have no idea whether Hitachi will deliver this feature, but I hope they do.

My only disappointment with Non-Disruptive Migrations is that it may have come to late for many organisations.  As virtualisation becomes more prevalent, migrations will be achieved using (for example) Storage vMotion in the hypervisor, negating the need to care whether the array can perform the migration on the host’s behalf.  However server virtualisation isn’t everywhere and the option of multi-tenancy is still a powerful one, even with the ability to virtualise the server.

Related Posts

• It’s About Time (and Money)
• No More Migration Tears

Disclaimer: Last year I attended Hitachi’s Influencer Forum in San Jose.  Hitachi paid for my travel and accommodation as well as most meals.  Most of the content of this event was NDA only and so hasn’t been discussed until now.  There is no requirement on me to blog about any of the content presented during the event.  I am not otherwise employed by Hitachi, or compensated for my time.

SCSI Reserve

First of all we need to take a step back and look at how LUN updates are serialised.  I/O integrity to a single LUN is achieved using the SCSI RESERVE command.  This allows an initiator (i.e. a host) to lock out a LUN from other updates.  This feature is especially necessary in clustered environments where a single LUN needs to be owned by a single cluster member during update.  Maintaining a reserve on a LUN prevents concurrent and conflicting updates from occurring (although strictly speaking updates can’t occur concurrently).  The “Reserve -> Update -> Release” process works well with environments where one or two hosts perform updates but in the virtual world where many hypervisors could be wanting to update large LUNs then the reserve process became a bottleneck.

SCSI reserve is essentially the same as the mainframe RESERVE macro, used to lock out a LUN to one specific address space and/or LPAR.  Unfortunately it’s rather a blunt tool in terms of managing data integrity and as volumes or LUNs increase in size, then the level of contention also increases, having a direct impact on performance and throughput.

The mainframe solution was to use Enqueues, essentially a “gentleman’s agreement” between the LPARs to share information on access at a more granular level.  Enqueues work on individual files and have a scope – they can be locked locally or globally.  This means a file can be reserved out within a single LPAR or across all LPARs sharing enqueue information.  LPARs implementing enqueues don’t then use RESERVE and so eliminate the performance overhead of constant volume reservation.

COMPARE and WRITE

VAAI within VMware uses the new COMPARE and WRITE command to perform a similar task to enqueues.  It enables data to be updated by one hypervisor by validating the contents are what was expected before the update.  The whole compare and write process has to happen as one operation (or atomically) to guarantee consistency.  The hypervisors confer with each other to maintain their “gentleman’s agreement” on who currently owns a virtual guest.

Virtual Machine Replication

Where does that leave us in terms of replicating virtual machines?  Well today replication is performed by the array, applying updates to the remote copy of a LUN in the order the updates were applied to the primary LUN.  What if an array manufacturer allowed both copies of a LUN to be read/write?  In this scenario hypervisors in two locations could update a single LUN over distance.  In order to maintain integrity, the COMPARE and WRITE command could be extended to include writing to the remote LUN as part of the single atomic operation.  The hypervisors then still retain their “gentleman’s agreement” on what’s being updated at any point in time.

Of course the above definition seems simple, but we have to consider latency and as distances increase, latency is likely to be the reason this solution breaks down.  Remote replication would need to be synchronous to guarantee integrity and so distance has a direct impact on performance.  Alternatively asynchronous replication could be used, with the understanding that data integrity isn’t completely guaranteed.

I’m looking forward to seeing how the vendors intend to tackle the LUN replication issue as it’s one we’ve all been waiting for, for a long time.

Firstly, let’s look at the VMware storage options.  You can choose to use block storage, in which case your LUN is formatted with the VMware proprietary VMFS file system.  Alternatively you can use NFS, in which case the NFS server manages the file system natively.  The VMFS option has a number of benefits; it allows VMware to tightly control the interaction of individual virtual machines and hypervisors, managing locking, thin provisioning and so on.  However there are also disadvantages.  Most storage arrays see LUNs as the smallest unit of presentable storage and perform operations (such as replication and cloning) at that level.  On non-virtualised environments multiple LUNs are usually presented to a host, so there’s no problem with those functions.  However with VMware the best practice is to use a smaller number of larger LUNs that encompass more than one virtual machine.  In effect this is a complete reverse of the physical host LUN model and can lead to issues when replicating or cloning on the array.

NFS provides a different approach.  NFS shares are presented to the hypervisor and virtual guests accessed simply as files.  NFS provides the locking mechanism and any additional functions, such as snapshots and replication.  As an extra bonus, the file system can still be accessed by traditional methods, making it possible to copy, edit and backup virtual machines away from the control of the hypervisor.  The trade-off between NFS and VMFS is seen as one of flexibility over performance but this is probably unfair and somewhat simplistic.

Where does VAAI come into this?  VAAI pushes some of the “heavy lifting” work required to clone virtual machines down to the array.  It also enables sub-LUN locking, improving performance.  In the discussion with Nigel he questioned why VMware were wanting to retain control via VMFS yet pushing functionality towards the array with VAAI.  I see this as a separation of control over execution.  VMware undoubtedly want to retain control of key storage functions within VMFS and the hypervisor.  If they do, they can charge for them.  It also makes sense for the hypervisor to be in control of tasks that directly affect the status of a running guest.  If VMware manages the replication process then data integrity can more easily be maintained.  However, copying data to and from the array through the hypervisor takes CPU cycles and network bandwidth.  It also doesn’t allow the array to execute in the most efficient manner.  By comparison, allowing the array to perform copy functions independently, data is not traversing the storage network, additional host cycles are not being used and the array can both schedule and execute the copy function much more efficiently as it is aware of the entire requirement, rather than seeing things in a sequential fashion.

Therefore we can see a dual benefit to VAAI; firstly the work is offloaded from the hypervisor, but second it is executed more efficiently.  In comparison to NFS, it seems that by implementing VAAI features, VMware are looking to retain even more control over the storage layer than before.

• Storage Choice. The single reference architecture uses AMS2500.  Why no solutions with VSP?  The VSP and VAAI support has been one of the most talked over things of the last few months from a virtualisation perspective, so I don’t see why only AMS is available.
• Hypervisor Choice. Speaking of VMware, the single hypervisor choice is Microsoft’s Hyper-V.  Again I’m confused by this.  VMware is by far the market owner in this space and many customers won’t even consider Hyper-V, regardless of how good it may be.  By not offering a VMware solution from day 1, it strikes me that HDS are cutting off a massive part of their available market.  Perhaps the logic was to fit a niche not currently covered by folks like VCE, but by only offering a single virtualisation layer HDS are doing exactly what VCE do today – delivering a single reference architecture.  Most of the virtualisation conversations I’ve had with HDS over the last few months have focused on demonstrating their close relationship with VMware.  This first release seems to ignore this.
• Reference Architecture. Speaking of reference architectures, from what I can see there is only a single configuration supported.  It isn’t easy to find on HDS’ website.  As far as I tell this page lists the configuration in the following PDF.  The reference architecture isn’t described as a product; with VCE it is at least clear that there are four products to choose from.  I can’t tell what HDS are selling; it may be a single fixed configuration; it may be flexible solutions based around a standard architecture of which the reference is an example – I’m just not sure.

No doubt it is a good thing that more choice is coming into the converged solutions market.  It would be good if HDS could clarify exactly what they are offering and to make those offering more comprehensive in terms of hypervisor, server and storage components.  If this isn’t done, then HDS will forever be playing catchup in a market place where the competitors have been active, have shipped product and most important, have gained experience and market share for quite some time.

Virsto

Virsto Overview

Virsto provide a software-only solution that works with Microsoft’s Hyper-V platform.  There are two versions available today; VSI for virtual servers and VDI for virtual desktop infrastructures.  The software works to address the issue all virtual solutions (and VDI in particular) have, that is the random nature of I/O.  Random I/O, especially random writes are more difficult to optimise as the storage array can’t efficiently use caching to minimise head movement on physical disks as well as they can with sequential writes.  The Virsto solution works by creating a virtual VHD object for the virtual machines.  As random writes are received, they are written sequentially to a log file in a similar fashion to a relational database.  The updates are then destaged asynchronously to the main logical copy of the VHD at a later stage.  Many virtual VHDs can be created from a single Virsto data and log LUN pair, making the solution ideal to deploy in a virtual environment.

Virsto seems like a simple but elegant answer to the issue of random I/O across many virtual guests.  Of course there’s no such thing as the proverbial free lunch and so there are considerations to make with this product.  Firstly, LUNs are created in the Virsto space using long GUIDs as identifiers.  The Virsto API is then used to manage these LUNs with more reasonable identifiers.  However drop to the standard Hyper-V dashboard and you’re back to meaningless GUID VHD names.  Next there’s the question of performance.  In a large environment, how will the Virsto driver handle workload prioritisation?  For example, I want to ensure my main Hyper-V guests are delivered the highest priority, but now all of my I/O is going through an additional layer on the hypervisor host.

Virsto is on test in the TSA Lab and I hope to have some feedback and more detail in a few weeks.

Atlantis

Atlantis Computing

The second solution comes from a company called Atlantis Computing.  Their ILIO (Inline Image Optimisation) product sits as a software virtual appliance between the VDI storage and the hypervisor.  The software takes advantage of the type of I/O performed by the Windows desktop, where much of the traffic is reading of common operating system files.  These don’t need to be served out of individual guest disk images but are managed by the ILIO virtual appliance directly, resulting in a significant reduction in I/O to the underlying disk storage.  Instead the appliance acts as a cache within the hypervisor.  Of course the immediate question here is data integrity and that could be a problem.  However if stateless desktops are deployed and the data for those desktops is stored elsewhere, then there’s no reason the solution should have data issues.  In fact, one of the benefits of using ILIO is that it can be deployed with local DAS disk rather than using SAN storage.  If the server crashes or the disk fails, then the guests can be restarted on another server.  This may be inconvenient for those users at the time, but in reality disk failures aren’t that common and in any case the stateless nature means no data should be lost.  Using ILIO addresses two of the issues with VDI – I/O performance and cost.  Using DAS for VDI is a much more compelling solution.  I’m hoping to get ILIO into the Lab too and give it a spin.

Either of these two startups could be acquired by Microsoft or VMware and be easily integrated into the base product.  As usual the boundary between storage and virtualisation continues to be blurred.

I’ve upgraded The Storage Architect lab servers to R2 SP1 and have been looking at how Dynamic Memory operates.  There are links at the end of this post that provide the official line, but I’ll be discussing my findings and thoughts here first.

Why Is Dynamic Memory needed?

Virtualisation is all about sharing resources – CPU, memory, disk space and so on, in a way that enables those resources to be used more efficiently.  When Windows servers were using 25% or less of their CPU capacity, there was an opportunity to make use of that wasted resource by stacking multiple Windows onto the same hardware using a hypervisor like VMware or Hyper-V.  The virtualisation approach is more suitable to Windows as the O/S doesn’t play well with multiple application installations in one Windows instance.  In addition, for maintenance and other management reasons, it may not be appropriate for multiple business applications to share the same server.

Sharing processor time is relatively easy and is already built into all operating systems that run multiple tasks – so pretty much everything since DOS days has been able to “timeslice”.  Memory however is a little more problematic.  All operating systems (bar a few like z/OS) assume that the memory they detect at boot time is there forever (or at least until the O/S shuts down).  This is a pretty reasonable assumption as hot plugging memory is something I don’t see most server hardware having; the cost of implementing electrical isolation on the motherboard slots doesn’t really justify the cost.

However memory is inevitably overallocated to hosts on their initial creation, even if it is subsequently fully utilised at some stage in the future.  Memory is also the main resource that virtualisation platforms never have enough of, so it makes sense to be able to be more efficient with its use.  Dynamic Memory for Hyper-V enables a host to manage the memory provided to the guests in a more efficient manner by dynamically increasing and decreasing the memory available to each guest.  However there are some caveats:

• The new feature requires the installation of SP1 on Windows Server 2008 R2 running Hyper-V (or Windows Hyper-V Server R2)
• Dynamic memory is only supported on Windows 7, Windows Server 2008 with SP2, Windows Server 2008 R2, Windows Vista with SP1, Windows Server 2003 R2 with SP2 and Windows Server 2003 with SP2 along with either SP1 for that platform or the SP1 version of the integration services.  See here for the specific details.
• Non-supported platforms will continue to use their full allocation of memory.

Integration Services is required on the guest O/S to fix the problem of notifying the O/S that the physical memory allocation has changed.  It looks like for Windows 7 and Windows Server 2008 SP2 that Service Pack 1 onwards effectively provides this capability and should be included in all future releases (although Microsoft would have to confirm that).

As you can imagine, adding and removing what appears to be physical memory isn’t a trivial task.  Remember that as far as the O/S is concerned, the memory it has access to is real physical memory.  Adding memory is no problem; the amount of apparent physical memory is simply increased.  However when taking memory back, the hypervisor needs to be aware that the memory slot may be in use.  In order to take it back from the guest O/S, the memory page would need to be released by paging the contents to disk or another memory page.  I don’t know the specifics of how Hyper-V performs dynamic memory management, but common sense would say that the pages to be retrieved should be the ones not in use by the guest.  Depending on how Windows views physical memory, it may not be possible to remove a non-sequential memory slot; there may be a requirement to remove memory from the end of the address space; it all depends on how Windows memory management algorithms work.  This is something I’ll speculate on after presenting the performance testing results.

Configuration

Dynamic Memory is specified using new options in guest Settings.  See the first screenshot.  Either Static or Dynamic is selected; for Static, a fixed amount of memory is selected; for Dynamic a Startup and Maximum figure is specified. A third parameter, Memory Buffer also needs to be set.  This represents an amount of memory Hyper-V attempts to reserve in addition to the in-use memory; so, if 1000MB have been assigned to a guest and the buffer is 20%, an additional 200MB is reserved.  There’s also an option to weight each guest in terms of priority; so less important virtual machines could be given lower priority when memory is low.

Testing Testing

The second screenshot shows my test environment.  This is a Hyper-V server with 8GB of RAM.  Initially memory is assigned to a number of Windows servers and two non-Windows servers; Memory Demand and Memory Status is shown for those guests running Integration Services.  Running the following command against one of the guests shows that the physical memory assumed by the guest is the initial value, in this case 512MB (screenshot 3).

get-wmiobject Win32_PhysicalMemory -computername hv41-win7-x64

This sort of makes sense, in that the physical memory slots seen at boot time is the value representing startup memory and so this value is also the minimum that the guest can have.  However as I found when testing, the reported memory assigned to a guest increases but never decreases.

To track memory usage, I wrote a small Powershell script that extracts the host and guest memory usage, displaying the results every 10 seconds.  The output is shown in the graph screenshot.  From a starting position of 512MB on the guest, I increased memory demand by starting Internet Explorer multiple times.  This increased memory figures for Guest Used and Physical.  It also increased the memory in use by the Host, all in step with each other.  Around 4 minutes into the test I started other guests on the host.  This immediately increased the Host Used memory sharply and as the Host became under pressure with memory, it caused the Guest’s assigned memory to drop to the bare minimum i.e. what was actually in use.  Once I relieved the pressure (around 9 minutes in) then started to close IE on the Guest, the memory figures returned to levels seen before the test.  There are some interesting pieces of information here:

• The Guest Physical MB figure increased from the base 512MB to towards the maximum assigned (in this case 1024MB).  Even when memory demand in the guest drops, this figure never reduces and is always a high watermark.  There could be a temptation to set the Maximum memory for each guest very high.  I can’t imagine this being wise as increased demand would start paging at the Hyper-V level, impacting all guests.
• At one stage I appeared to have more memory in use than Assigned (around 8 minutes in).  I’m not sure how this could happen, other than for Hyper-V to be paging some of the active guest memory to disk.

Memory usage within Windows was never clear in the first place; Dynamic  Memory just makes things more complex; correctly determining memory usage will require information from both the hypervisor and the guest.  I’m hoping to develop a tool that will make the process of understanding the configuration an easier one.  In the meantime I’d be interested in feedback on how other people have tested this new feature.